↓ Skip to main content
  1. Posts/

Nothing Listening: How Coding Agents Helped Me Close My Database's Front Door

Keith Hodo
Author
Keith Hodo
Solutions Architect at AWS. Writing about cloud, agentic AI, and the journey.
Table of Contents

Cascadian Gamers is a collective of friends from Portland, Seattle, and Vancouver who come together every year to raise money for children’s hospitals under the Extra Life banner. We game, we run a raffle for donors, and over the years the raffle has grown its own software. A few of us maintain the tech side. I am the only one writing the code, and the others are the technical stakeholders who keep me honest.

I don’t expect anyone to donate (my content is intentionally free for anyone), but if you feel compelled, my Extra Life page is here. Donations to me support Seattle Children’s Hospital, and 100% of proceeds go directly to helping sick kids and their families.

Here is what the software does today. Donations come in from the DonorDrive API, Lambda ETL jobs move the data along, and SQL Server on Amazon RDS sits inside the VPC holding everything the admin site and our AI agents on Bedrock AgentCore depend on. I try to release to production every two weeks, as time allows. Every so often a release needs my hands on the database directly. A migration needs running. Numbers need reconciling. An isolation setting needs checking before I trust a change.

An empty Port of Seattle container terminal at sunset, cranes idle and the asphalt still wet from rain
An empty Port of Seattle at sunset. Nothing coming in, nothing listening. That is how I want my ports.

What Database Maintenance Used to Look Like
#

Before AWS, everything lived in Azure, and I connected SQL Server Management Studio straight from my own machine. That was convenient. It also meant the database was not in a private subnet, and anyone with the URL could address it. I would not call that a security risk, but it was more open than what we have in AWS today.

When we moved to AWS between 2023 and 2024, the database went into a private subnet and I built a Windows jump box in each environment to reach it. Doing any database maintenance meant logging into AWS, turning on the jump box, updating the RDP file with its new address, making sure my current IP address was in the security group, and only then opening SSMS to make the changes, which carry their own level of tedium. Each box had a public address while it ran and sat in the VPC default security group. None of it was in infrastructure as code, and the boxes did not even have the AWS CLI.

That last part bit me during this week’s production release. I needed a password from Secrets Manager, and the box could not fetch it, so I copied it out of the console by hand. In the same release I found that the read-only SQL login behind VAR, the agent that answers database questions for the rest of our AI tooling, had quietly stopped working. SQL Server creates logins with password expiration on by default, the password had aged out, and every call VAR made failed at login with error 18487. The metrics had no successful VAR queries going back to the start of October. Nothing alarmed. I reset the password, turned expiration off for that login, and opened an issue for the alarm that should have caught it.

How I Found the Better Path
#

None of this is new. Session Manager has supported port forwarding to remote hosts for a while. I found it because I started pointing newer frontier models from Anthropic and OpenAI at my codebase, specifically my CDK, and asking how to make the infrastructure more secure. I ranked what they found from high to medium to low and started working down the list. Getting rid of the public jump boxes was one of those findings.

Older models had looked at the same code and told me I was following best practices. That was a fair read of yesterday’s bar. The newer models hold it to a higher one.

Access Before Encryption
#

The original plan was to encrypt the database at rest first and rebuild the jump boxes last. I reversed that order. Encryption at rest protects stored copies: storage, automated backups, and snapshots. It is still on the plan. For a database that runs all day, though, the route people and software use to reach it was something I could improve right away. Operator access went first, then the database network perimeter, then encryption at rest.

It also turned out I could drop the desktop entirely. Everything I do on release day needs a SQL client and queries someone can review. With sqlcmd, I can record the exact command that produced a result and reuse it at the next release. That is a lot easier to put in a runbook than a schema change I typed into SSMS on a remote desktop.

The Operator Relay
#

Before: a public Windows jump box, reached over RDP from the internet.

flowchart LR
  A["💻 Operator laptop
RDP client"] -->|"RDP"| B["🌐 Internet"] B -->|"public IP
+ my IP in the SG"| C subgraph VPC1 ["VPC"] subgraph PUB ["Public subnet"] C["🪟 Windows jump box
public IP
default security group"] end subgraph PRIV1 ["Private subnet"] E[("RDS
SQL Server")] end end C -->|"SSMS
TCP 1433"| E classDef operator fill:#1e3a5f,stroke:#60a5fa,color:#f8fafc classDef exposed fill:#5f1e1e,stroke:#f87171,color:#f8fafc,stroke-width:2px classDef db fill:#1e293b,stroke:#94a3b8,color:#f8fafc classDef internet fill:#5f1e1e,stroke:#f87171,color:#f8fafc class A operator class B internet class C exposed class E db style PUB fill:transparent,stroke:#f87171,stroke-dasharray:5 5,color:#fca5a5 style PRIV1 fill:transparent,stroke:#4ade80,stroke-dasharray:5 5,color:#86efac style VPC1 fill:transparent,stroke:#94a3b8,color:#cbd5e1

After: a private Linux relay, reached through Session Manager with nothing listening inbound.

flowchart LR
  F["💻 Operator terminal
AWS CLI + sqlcmd"] -->|"IAM-authorized
session over TLS"| G["🔐 Session
Manager"] G <-->|"agent connects
outbound"| H subgraph VPC2 ["VPC"] subgraph PRIV2 ["Private subnet"] H["🐧 Linux relay
no public IP
no inbound rules"] I[("RDS
SQL Server")] end end H -->|"TCP 1433"| I classDef operator fill:#1e3a5f,stroke:#60a5fa,color:#f8fafc classDef aws fill:#5c3d0e,stroke:#fbbf24,color:#f8fafc classDef safe fill:#14532d,stroke:#4ade80,color:#f8fafc,stroke-width:2px classDef db fill:#1e293b,stroke:#94a3b8,color:#f8fafc class F operator class G aws class H safe class I db style PRIV2 fill:transparent,stroke:#4ade80,stroke-dasharray:5 5,color:#86efac style VPC2 fill:transparent,stroke:#94a3b8,color:#cbd5e1

AWS calls this Session Manager port forwarding to a remote host, and introduced it in an announcement on the Management and Governance blog. I call the EC2 instance the operator relay. The AWS CLI and the Session Manager plugin open a listener on my machine, and the SSM agent on the instance forwards the connection to RDS on TCP 1433. I checked the listener with ss -ltn and confirmed it was bound to loopback only.

The relay is an Arm-based Amazon Linux 2023 t4g.micro in a CDK nested stack. I picked the micro over the nano to leave headroom for agent self-updates and a couple of simultaneous tunnels. It lives in a private subnet with no public address, no key pair, and no user data. It requires IMDSv2 and has an encrypted 8 GB gp3 root volume. Its security group has no inbound rules at all. Outbound, it allows HTTPS and SQL traffic to the database security group on TCP 1433, and the database admits the relay by security group reference.

The instance role uses AmazonSSMManagedInstanceCore plus an explicit deny on ssm:GetParameter* for the application’s Parameter Store paths. If someone does get a shell on the relay, that role cannot read the application’s connection strings. The stack also defines a system status check alarm wired to the EC2 recover action.

CDK gets the AMI from a public Systems Manager parameter, which means CloudFormation can resolve a newer image on a later stack update. A new AMI being published does nothing to the running instance on its own. Patching is handled by a weekly AWS-RunPatchBaseline association with Install and RebootIfNeeded, limited to its scheduled slot. Either a stack update that replaces the instance or a patch reboot will drop open tunnels, so I plan around both.

Walkthrough: SQL Server on RDS Through a Session Manager Relay
#

This is the workflow I run, with placeholders in angle brackets. <rds-endpoint> is the actual RDS endpoint name and <loopback-address> is the standard IPv4 localhost address. The examples use Oregon (us-west-2) and my develop port.

What the workflow needs

  • AWS CLI v2 with credentials allowed to call ssm:StartSession on the relay and use the port-forwarding document. The status check and the password lookup also need ssm:DescribeInstanceInformation and secretsmanager:GetSecretValue.
  • The Session Manager plugin.
  • sqlcmd from mssql-tools18, with Microsoft ODBC Driver 18 for SQL Server.
  • The regional RDS CA bundle in the operating system trust store.
  • An SSM-managed relay that can reach the database on TCP 1433, defined in step 1.

1. Define the relay in CDK
#

This excerpt sits inside the nested stack constructor, with the VPC passed in from the parent. DatabaseNetwork.AllowSqlServer is a helper in my repo that adds TCP 1433 egress to the database security group and the matching ingress from the relay group. The key pair and user data are deliberately left unset.

SecurityGroup = new SecurityGroup(this, "RelaySecurityGroup", new SecurityGroupProps
{
    Vpc = vpc,
    AllowAllOutbound = false,
});
SecurityGroup.AddEgressRule(Peer.AnyIpv4(), Port.Tcp(443));
DatabaseNetwork.AllowSqlServer(SecurityGroup, DatabaseNetwork.ImportSecurityGroup(this), "operator relay");

var role = new Role(this, "RelayRole", new RoleProps
{
    AssumedBy = new ServicePrincipal("ec2.amazonaws.com"),
    ManagedPolicies = new[] { ManagedPolicy.FromAwsManagedPolicyName("AmazonSSMManagedInstanceCore") },
});
role.AddToPolicy(new PolicyStatement(new PolicyStatementProps
{
    Effect = Effect.DENY,
    Actions = new[] { "ssm:GetParameter", "ssm:GetParameters", "ssm:GetParametersByPath", "ssm:GetParameterHistory" },
    Resources = new[] { FormatArn(new ArnComponents { Service = "ssm", Resource = "parameter", ResourceName = "myapp/*" }) },
}));

Relay = new Instance_(this, "Relay", new InstanceProps
{
    Vpc = vpc,
    VpcSubnets = new SubnetSelection { SubnetType = SubnetType.PRIVATE_WITH_EGRESS },
    InstanceType = InstanceType.Of(InstanceClass.T4G, InstanceSize.MICRO),
    MachineImage = MachineImage.LatestAmazonLinux2023(new AmazonLinux2023ImageSsmParameterProps
    {
        CpuType = AmazonLinuxCpuType.ARM_64,
        Kernel = AmazonLinux2023Kernel.KERNEL_6_1,
    }),
    SecurityGroup = SecurityGroup,
    Role = role,
    AssociatePublicIpAddress = false,
    RequireImdsv2 = true,
    BlockDevices = new[] { new BlockDevice { DeviceName = "/dev/xvda", Volume = BlockDeviceVolume.Ebs(8,
        new EbsDeviceOptions { Encrypted = true, VolumeType = EbsDeviceVolumeType.GP3, DeleteOnTermination = true }) } },
});

The same stack holds the weekly patch association and the StatusCheckFailed_System recovery alarm. I trimmed those from the excerpt to keep it readable.

2. Install the client tools
#

I run all of this on an internal developer desktop, an x86-64 Amazon Linux 2023 machine, and it is also where I run Kiro Crew. Kiro Crew is an open source AI workspace where agents keep working across sessions, remember my corrections, and run scheduled or event-driven tasks. More on how it fits in later. The install uses Microsoft’s RHEL 9 repository. On macOS, the equivalents are the AWS Session Manager plugin instructions and Microsoft’s macOS ODBC and tools instructions.

sudo dnf install -y https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm
curl -fsSL https://packages.microsoft.com/config/rhel/9/prod.repo | sudo tee /etc/yum.repos.d/mssql-release.repo
sudo ACCEPT_EULA=Y dnf install -y mssql-tools18 unixODBC
sudo curl -fsSL -o /etc/pki/ca-trust/source/anchors/rds-us-west-2-bundle.pem https://truststore.pki.rds.amazonaws.com/us-west-2/us-west-2-bundle.pem
sudo update-ca-trust

3. Check the relay is managed
#

The AWS announcement checks the instance by opening an interactive session. I query its management status instead, because an operator who only forwards ports may not have shell access at all. The output should say Online.

aws ssm describe-instance-information --profile '<aws-profile>' --region us-west-2 --filters 'Key=InstanceIds,Values=<relay-instance-id>' --query 'InstanceInformationList[].PingStatus' --output text

4. Map the endpoint name to loopback
#

ODBC Driver 18 encrypts by default and validates the server certificate. Connecting to a loopback address would fail hostname validation, so I map the RDS endpoint name to loopback instead. The connection goes through the tunnel and the certificate check still runs all the way to RDS. The grep guards against a duplicate entry. This mapping affects every other tool on the machine until it comes back out.

grep -qF ' <rds-endpoint>' /etc/hosts || echo '<loopback-address> <rds-endpoint>' | sudo tee -a /etc/hosts

5. Open the tunnel in terminal 1
#

I use the CLI shorthand for --parameters; the announcement uses JSON for the same values. Expect Port 11433 opened... followed by Waiting for connections.... It waits in the foreground by design, and Ctrl-C closes it. I use 11433 for develop and 11434 for production so I am less likely to mix them up. The port is a hint, though. The endpoint and relay ID still have to match the environment.

aws ssm start-session --profile '<aws-profile>' --region us-west-2 --target '<relay-instance-id>' --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters 'host=<rds-endpoint>,portNumber=1433,localPortNumber=11433'

6. Check the listener
#

From another terminal, the first command is for Linux and the second for macOS. The listener should be on the IPv4 loopback address, port 11433. A wildcard listener would be reachable from other network interfaces on the machine.

ss -ltn | grep 11433
lsof -nP -iTCP:11433 -sTCP:LISTEN

7. Query with sqlcmd in terminal 2
#

The login’s password lives in Secrets Manager. SQLCMDPASSWORD keeps it out of process arguments and shell history. The -I flag matters more than it looks. SSMS turns QUOTED_IDENTIFIER on by default, and sqlcmd does not. One of our tables has a filtered unique index, and any write to it without that setting fails with error 1934. I never saw that error in years of SSMS, and the runbook keeps -I on every command so I never have to. For a script I swap -Q for -i file.sql, and for an interactive session I drop -Q and submit batches with GO. Temporary tables and transactions persist for the life of that connection. GUI clients such as SSMS, Azure Data Studio, and DBeaver work too, using the same endpoint name and local port with certificate validation left on.

export SQLCMDPASSWORD="$(aws secretsmanager get-secret-value --profile '<aws-profile>' --region us-west-2 --secret-id '<secret-id>' --query SecretString --output text)"
/opt/mssql-tools18/bin/sqlcmd -S '<rds-endpoint>,11433' -U '<login>' -d '<database>' -I -Q 'SELECT @@SERVERNAME'
unset SQLCMDPASSWORD

8. Optionally restrict operators to port forwarding
#

In my develop deployment, only admin-class roles can open sessions at all. If I hand access to a dedicated operator identity later, this is the shape of the identity policy I would start from. It allows the relay and the AWS-owned forwarding document only, and the session resource tag limits resume and terminate to sessions the caller started. Any other grants that identity holds still need checking for broader session permissions. The example assumes the commercial AWS partition. See the Session Manager policy examples and the session document access check.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:StartSession",
      "Resource": [
        "arn:aws:ec2:us-west-2:<account-id>:instance/<relay-instance-id>",
        "arn:aws:ssm:us-west-2::document/AWS-StartPortForwardingSessionToRemoteHost"
      ],
      "Condition": { "BoolIfExists": { "ssm:SessionDocumentAccessCheck": "true" } }
    },
    {
      "Effect": "Allow",
      "Action": ["ssm:TerminateSession", "ssm:ResumeSession"],
      "Resource": "arn:aws:ssm:us-west-2:<account-id>:session/*",
      "Condition": { "StringLike": { "ssm:resourceTag/aws:ssmmessages:session-id": "${aws:userid}" } }
    }
  ]
}

9. Close up
#

Ctrl-C in terminal 1 closes the tunnel. In terminal 2 I clear the password variable and confirm it is gone without printing its value. The hosts-file entry comes out when other tools need to reach the endpoint directly again.

unset SQLCMDPASSWORD
test -z "${SQLCMDPASSWORD+x}" && printf '%s\n' 'SQLCMDPASSWORD is unset'

Troubleshooting
#

SymptomLikely causeFix
TargetNotConnectedThe agent is offline, rebooting, or cannot reach Systems Manager over TCP 443Check managed status, then HTTPS egress and routing
Session opens but sqlcmd times outWrong local port, missing relay egress on TCP 1433, or missing database ingress from the relay groupCheck the listener and the matching security group rules
Certificate verification failsThe RDS CA is missing, or the client is connecting by addressTrust the regional CA and connect by endpoint name
Error 18456 or 18487Login failed (bad credential), or the password expiredCheck the login and secret. An expired password needs a reset by its owner, plus updates to every consumer
Error 1934QUOTED_IDENTIFIER is off for a write that requires itKeep -I on the sqlcmd invocation

What Got Better
#

The relay has no public address, no inbound rules, no RDP path, and no SSH key pair. IAM authorizes ssm:StartSession, and CloudTrail records the call, so I can tie every session to an identity even when an operator changes networks. The SQL egress rule is scoped to one port and one destination security group, and the access and patch settings are reviewed as code instead of living in someone’s memory of how the box was built.

Session permissions matter as much as the network does. A principal allowed to start sessions without a document restriction can open a shell just as easily as a tunnel. In develop, only admin-class roles can open a session. If non-admin operators get access later, their policy should pin the document to port forwarding. I left Session Manager session logging off in develop, where access is admin-only and the forwarded traffic is SQL, and I will revisit that for production. CloudTrail records session starts either way. Session Manager does not log the contents of port-forwarding sessions.

What It Costs
#

To compare like for like, I priced 730 hours of shared-tenancy On-Demand use in Oregon (us-west-2), excluding tax, credits, burst CPU charges, data transfer, and any new network infrastructure. I usually stopped the old boxes between uses, so continuous runtime is a common baseline for something I can leave running, and not what the jump boxes actually cost me.

ItemAWS unit priceMonthly calculation
Windows t3.large, license included$0.1108/hour730 × $0.1108 = $80.88
Public IPv4$0.005/hour730 × $0.005 = $3.65
Linux t4g.micro$0.0084/hour730 × $0.0084 = $6.13
gp3 root volume$0.08/GB-month8 × $0.08 = $0.64

That comes to $84.53 a month for Windows with its address, before disk, and $6.77 for the relay with its disk. The Windows number was higher than my earlier $60 to $70 guess. Session Manager adds no charge for accessing EC2 instances. Prices are from EC2 On-Demand pricing, EBS pricing, VPC pricing, and Systems Manager pricing, checked November 20, 2026.

The bigger saving is time. No more starting an instance, waiting for RDP, and maintaining a Windows image I only touched on release day. The relay rides on an existing NAT gateway shared with other workloads. If I had needed private connectivity just for the relay, it would belong in the comparison. Interface endpoints for ssm and ssmmessages (plus ec2messages where agent compatibility requires it) are the alternative, and they carry their own hourly charges.

Where Coding Agents Fit
#

My coding agents have a shell on a developer machine, and that changes the question I ask about database access. I still need to know who can open a connection. I also need to know what a shell can reach and whose credentials it would use. An agent helping prepare a migration has enough context to write a very convincing command. I want the person responsible for the migration deciding whether it runs.

The rule today is simple. The agent never opens a tunnel and never holds a database password. In Kiro Crew, the agent’s runtime policy denies the aws ssm command family and CloudFormation mutations. It prepares commands and queries, including the checks I want before a migration. I review them, run them from my own terminal, and paste the relevant results back. The agent checks those results against what we expected and writes the record. Production writes stay with a human.

Take the isolation settings check before a release. The agent drafts a read-only query and explains which output would change the release plan. I review it, run it through the relay, and hand back the setting. That handoff has a concrete input and output that fits in a runbook. The old version was me logging into the Windows jump box, going through the whole routine described earlier, and making the schema updates by hand.

Kiro Crew gets the longer repository work, the kind with a reviewable result. It deserves its own post, so I will keep this part short. For a schema change, I ask it to draft the migration, find the stored procedures and callers that depend on the affected columns, and write the preflight and verification queries. For a stored procedure, I ask it to preserve the caller contract, add tests for the edge cases, and describe how to verify the change in develop. Either way, what comes back is a diff and a set of checks I can review before anything touches RDS.

Local coding agents are better when the work needs a quick round trip. I ask for a reconciliation query that returns aggregate counts, review the joins and filters, run it myself, and hand back the counts for interpretation. If a migration check comes back wrong, the agent traces the relevant code and proposes the next query while I keep control of the session. Database passwords and donor records stay out of that exchange.

The short-lived tunnel also gives me an obvious start and stop. Loopback keeps the listener local, but any local process can reach it while it is open, which is one reason the database login and its permissions still matter. IAM decides which identity can open the session. A prompt can tell an agent the rule, but enforcement has to live in a control the agent cannot change. A long-lived public jump box with stored credentials would put far more trust in every process that could reach it.

Eventually I may let an agent issue read-only queries through a tunnel I open. That would need a dedicated read-only SQL login and a credential path built for the purpose, with an owner, a rotation plan, and an alarm on login failures. I have already seen what happens when a machine login expires and nobody notices. Any access I add needs a way to tell me it still works.

What Comes Next
#

The develop relay went through CI. I checked the deployed stack, confirmed there was no public address and no inbound rule, and watched the SSM agent check in. I queried the server name and a row count through the tunnel with certificate validation on. I simulated session authorization across every non-service principal in develop and confirmed only admin-class roles could open a session. Stopping the old develop box released its public address and closed its public RDP path.

The change also went through review for security, infrastructure, test quality, and maintainability. I deliberately broke settings like public addressing, IMDSv2, disk encryption, egress, the deny policy, and the patch schedule, and made sure the tests caught each one.

The production relay shipped last night, and it works. The old production jump box stays stopped until the last of its security group rules come out and I terminate it. After that, I tighten the database’s remaining network paths, and then encryption at rest finally gets its turn.

The bigger lesson for me has little to do with Session Manager. A model telling me my infrastructure followed best practices yesterday does not mean it clears today’s security bar. So I keep pointing new models at old code, ranking what they find, and fixing it in order. Curiosity has always been the job in security and engineering, and with agentic coding there has never been a better time to learn something new.

The Cascadian Gamers team page lives on Extra Life. Donations there support Children’s Miracle Network Hospitals, and donations to me support Seattle Children’s Hospital. Either way, the database they land in is a little better protected than it was last week.

If you like what you’re reading, or it’s helpful, please consider donating! 100% of proceeds go directly to helping sick kids and their families.

Keith

Sources
#

Prices were checked November 20, 2026, for Oregon (us-west-2), shared tenancy, On-Demand.

Pricing

Session Manager and IAM

Agent tooling

  • Kiro Crew, the open source agent workspace described in the coding agents section

SQL Server tooling

How the walkthrough was adapted

  • The CDK excerpt comes from the project’s relay stack, trimmed, with constructs renamed and the real Parameter Store prefixes replaced by myapp/*.
  • The install, tunnel, and sqlcmd commands come from the project runbook, parameterized. The runbook also documents a certificate validation bypass. The walkthrough leaves validation on throughout.
  • Only the Amazon Linux 2023 developer desktop setup was tested. The macOS listener check and the GUI clients are listed as alternatives I have not validated.