Skip to main content

Nexgen V3 databases

The Nexgen apps store data in two AWS database services: MySQL on RDS and MongoDB on DocumentDB. This page records what runs where, the parameter settings that must be kept, and the procedure for a DocumentDB major version upgrade, which was last run in October 2026 to move both clusters from 5.0.0 to 8.0.2.

Everything is in AWS account 202077789656, region us-east-1, under the Copilot app nexgen-v3.

What uses the databases​

  • nexgen-backend (the paid app) and nexgen-free-backend (the trial app) each have their own MySQL database and share one MongoDB database.
  • The mcp-server and api services read from both.
  • The V2 nexgen, nexgen-free, and nexgen-api services run against the same databases as V3. V2 is PHP 5.6 with version 1.7.5 of the mongodb extension, so any database change has to keep those older clients working.

MySQL​

MySQL runs on RDS as MySQL 8.4 with utf8mb4. Each environment has one instance holding the databases for both apps.

  • Instances: nexgen-v3-prod-mysql and nexgen-v3-staging-mysql.
  • Databases: nexgen for the paid app and nexgen_free for the trial app.
  • Instance class: db.t3.medium on production and staging. Staging matches production so that the staging performance tests measure what customers get.
  • Storage: gp3, encrypted, 50 GB growing to 200 GB on production and 20 GB growing to 50 GB on staging.
  • Users: npuser is the read and write user for the apps. mcp_readonly is the read-only user for the mcp-server, and flask_readonly is the read-only user for the api service.
  • Deletion protection is on, and the instance is not publicly accessible.
  • Template: copilot/environments/addons/rds-mysql.yml in the nexgen_v3 repo.

MySQL settings that must stay​

The parameter group (family mysql8.4) carries overrides that the apps depend on. Keep all of them through any upgrade or parameter group change.

  • sql_mode: NO_ENGINE_SUBSTITUTION. The Laravel apps run with strict mode off. Losing this override turns on the 8.4 defaults, including STRICT_TRANS_TABLES and ONLY_FULL_GROUP_BY, which breaks legacy queries and date defaults.
  • authentication_policy: "*:mysql_native_password". The V2 PHP 5.6 clients abort on the 8.4 default caching_sha2_password handshake, even for accounts that use the native plugin.
  • character_set_server: utf8mb4 and collation_server: utf8mb4_unicode_ci.

Because strict mode is off, MySQL stores a bad value without raising an error. For example, text written to an int column is stored as 0.

MySQL version history​

MySQL was upgraded from 8.0 to 8.4 in July 2026, because RDS standard support for 8.0 ended on 2026-07-31. The template sets AllowMajorVersionUpgrade: true, so CloudFormation performed that upgrade in place through an environment deploy. MySQL 9 is not offered on RDS, and the V2 clients would block it in any case.

DocumentDB (MongoDB)​

MongoDB runs on Amazon DocumentDB, which is MongoDB-compatible and is not the MongoDB server itself. Some aggregation stages that work on a local MongoDB are rejected by DocumentDB, and its query planner can choose a different index.

  • Clusters: nexgen-v3-prod-docdb and nexgen-v3-staging-docdb, each with one db.t3.medium instance.
  • Database: nexgen_mdb, shared by the paid app and the trial app. There is no separate trial MongoDB database.
  • Engine version: 8.0.2 on both clusters.
  • Parameter groups (family docdb8.0, owned by CloudFormation): docdbparametergroup-6f6vxktnowdu on production and docdbparametergroup-ycatcilubs6z on staging.
  • Storage is encrypted and deletion protection is on.
  • Template: copilot/environments/addons/documentdb.yml in the nexgen_v3 repo.

DocumentDB settings that must stay​

Keep planner_version at 1.0

The docdb8.0 family defaults planner_version to 3.0. On staging, that planner read the whole foods collection to build the recent ingredients list for small accounts. Read IOPS reached 11,000, CPU reached 87 percent, and every ingredient picker on the trial site hung. Never accept an 8.0 parameter group that leaves planner_version at the family default.

  • planner_version: "1.0". This is the planner the 5.0 clusters ran. The recent-list queries also hint the owner_1_Recipe_1_FoodID_-1 index as a second defense.
  • tls: disabled. Every client connects with a plain mongodb:// connection string, on production as well as staging. A client that passes tls=true gets a connection reset.

To read the two values from a parameter group:

aws docdb describe-db-cluster-parameters --region us-east-1 --db-cluster-parameter-group-name <group> --query "Parameters[?ParameterName=='tls'||ParameterName=='planner_version'].[ParameterName,ParameterValue]" --output text

DocumentDB version history​

Both clusters were upgraded in place from 5.0.0 to 8.0.2. There is no DocumentDB 7.0, so 5.0.0 upgrades straight to 8.0.2.

  • Staging: 2026-09-30, about 13 minutes offline.
  • Production: 2026-10-04, offline from 19:32:41 to 19:43:21 UTC, which is 10 minutes 40 seconds. A 212-second index metadata refresh followed with the cluster online.

On production, the instance still reports pending-reboot for its parameter group after three reboots and a re-association of the group. The running values match the group's values, so nothing is wrong with the cluster, and the flag was left for AWS support. Until it clears, do not rely on DBClusterParameterGroupStatus on production to confirm that a later parameter change has landed. Read the parameter values instead.

The production upgrade left two snapshots from 2026-10-04: the manual preupgrade-nexgen-v3-prod-docdb-5-0-0-to-8-0-2-<date> snapshot and one that AWS took itself with a time suffix. Both are kept until someone decides production is settled and deletes them.

Backups​

  • Automated backups are kept for 7 days on production and 1 day on staging, for both MySQL and DocumentDB.
  • Production also has the nexgen-v3-prod-db-backup-vault AWS Backup plan, which covers the MySQL instance and the DocumentDB cluster. It takes a daily backup at 02:00 UTC kept for 7 days, and a monthly backup on the first of the month at 03:00 UTC kept for a year. The plan is defined in copilot/environments/addons/db-backup.yml and exists only on production.

To copy production data into staging, see Refreshing staging from production.

Connecting​

Neither database is reachable from outside its VPC. Connect through the bastion service of the same environment, which has the mysql client and mongosh installed and the connection details in its environment variables.

copilot svc exec --app nexgen-v3 --env staging --name bastion

Then, inside the bastion:

mysql -h $DB_HOST -u $DB_USERNAME -p"$DB_PASSWORD" nexgen
mongosh "mongodb://$MDB_USERNAME:$MDB_PASSWORD@$MDB_HOST:27017/$MDB_DATABASE?authSource=$MDB_AUTH_DB&retryWrites=false&directConnection=true"

The app passwords are SSM parameters under /copilot/nexgen-v3/<env>/secrets/ (DB_PASSWORD, DOCDB_PASSWORD, MCP_DB_PASSWORD). The master passwords are in Secrets Manager.

Checking versions and status​

These commands only read.

aws rds describe-db-instances --region us-east-1 --query 'DBInstances[].[DBInstanceIdentifier,DBInstanceStatus,Engine,EngineVersion]' --output table
aws docdb describe-db-clusters --region us-east-1 --db-cluster-identifier nexgen-v3-prod-docdb --query 'DBClusters[0].{version:EngineVersion,status:Status,group:DBClusterParameterGroup,backupDays:BackupRetentionPeriod,maintenance:PreferredMaintenanceWindow,pending:PendingModifiedValues,members:DBClusterMembers[].{id:DBInstanceIdentifier,groupStatus:DBClusterParameterGroupStatus}}'

Upgrading DocumentDB to a new major version​

AWS::DocDB::DBCluster has no way to pass the major version upgrade flag, so the AWS CLI drives the upgrade and the CloudFormation template catches up afterwards. Upgrade staging first and leave it running for a few days before production.

Do not run an environment deploy while the template is ahead of the cluster

If documentdb.yml names a newer engine version than the cluster runs, copilot env deploy makes CloudFormation attempt the major version change, which fails and rolls back. The production deploy workflow never runs an environment deploy, so this only matters for commands run by hand. As of 2026-10-01 the test environment's cluster had not been upgraded, so an environment deploy there fails this way until it is.

There is no in-place downgrade. The only way back is to restore the pre-upgrade snapshot into a new cluster, which loses every write made after the snapshot.

In the commands below, replace <env> with staging or prod, and the version numbers with the versions involved. The examples use the 5.0.0 to 8.0.2 values.

1. Bring the backends to the tested build​

Production must run the same backend build that was verified against the new engine on staging. Deploy that image tag to nexgen-backend and nexgen-free-backend with the Deploy to Prod workflow, then run the migrations by hand as described in Nexgen V3 deployment. Confirm with:

curl -s https://app.nutritionistpro.com/version
curl -s https://trial.nutritionistpro.com/version

2. Pre-flight checks​

Run the describe command from Checking versions and status, then check the current parameter values and the upgrade targets:

aws docdb describe-db-cluster-parameters --region us-east-1 --db-cluster-parameter-group-name <current group> --query "Parameters[?ParameterName=='tls'].[ParameterName,ParameterValue]" --output text
aws docdb describe-db-engine-versions --region us-east-1 --engine docdb --engine-version 5.0.0 --query 'DBEngineVersions[0].ValidUpgradeTarget[].EngineVersion'

Expect status available, nothing pending, tls disabled, and the target version in the list. Stop if tls is anything other than disabled, because the new parameter group must match what the cluster runs today.

Pick the window. Allow about 15 minutes during which the paid site, the trial site, the MCP server, and the customer API cannot reach MongoDB. Chargebee retries failed webhooks.

3. Take a snapshot and create the parameter group​

Neither step causes downtime. Take the snapshot as close to the window as possible, since it is the rollback point.

aws docdb create-db-cluster-snapshot --region us-east-1 --db-cluster-identifier nexgen-v3-<env>-docdb --db-cluster-snapshot-identifier preupgrade-nexgen-v3-<env>-docdb-5-0-0-to-8-0-2-<yyyymmdd>
aws docdb describe-db-cluster-snapshots --region us-east-1 --db-cluster-snapshot-identifier preupgrade-nexgen-v3-<env>-docdb-5-0-0-to-8-0-2-<yyyymmdd> --query 'DBClusterSnapshots[0].[Status,PercentProgress]'
aws docdb create-db-cluster-parameter-group --region us-east-1 --db-cluster-parameter-group-name nexgen-v3-<env>-docdb80-manual --db-parameter-group-family docdb8.0 --description "nexgen-v3-<env> DocumentDB 8.0 manual (TLS disabled, planner 1.0)"
aws docdb modify-db-cluster-parameter-group --region us-east-1 --db-cluster-parameter-group-name nexgen-v3-<env>-docdb80-manual --parameters "ParameterName=tls,ParameterValue=disabled,ApplyMethod=pending-reboot" "ParameterName=planner_version,ParameterValue=1.0,ApplyMethod=immediate"
aws docdb describe-db-cluster-parameters --region us-east-1 --db-cluster-parameter-group-name nexgen-v3-<env>-docdb80-manual --query "Parameters[?ParameterName=='tls'||ParameterName=='planner_version'].[ParameterName,ParameterValue]" --output text

Do not continue until the snapshot is available and the last command prints tls disabled and planner_version 1.0.

4. Upgrade​

This step is the outage.

aws docdb modify-db-cluster --region us-east-1 --db-cluster-identifier nexgen-v3-<env>-docdb --engine-version 8.0.2 --db-cluster-parameter-group-name nexgen-v3-<env>-docdb80-manual --allow-major-version-upgrade --apply-immediately
aws docdb describe-events --region us-east-1 --source-identifier nexgen-v3-<env>-docdb --source-type db-cluster --duration 60

Repeat the events command. The "Taking database cluster offline" event and the return to available bracket the downtime.

5. Verify​

  • The describe command shows the new version, status available, the hand-made group, and in-sync.
  • Both /version endpoints answer.
  • Log in to the paid site, search ingredients, open a recipe, and open a label preview. On the trial site, open an ingredient picker on a small account. That picker runs the recent-list query that hung on staging under planner_version 3.0.
  • Check that the MCP server, the customer API, and the V2 /version and /health paths answer. If the api or mcp-server service holds a dead connection, redeploy it with its current image tag.
  • On staging, run the nexgen-workflow-staging-cluster and nexgen-free-workflow-staging-cluster Playwright projects.
  • Watch ReadIOPS, CPUUtilization, and BufferCacheHitRatio in the AWS/DocDB CloudWatch namespace for the first half hour. Sustained read IOPS in the thousands is the sign of the planner regression, so check planner_version first.

A performance problem after the upgrade is fixed forward, through parameter group values or index hints.

6. Align CloudFormation and clean up​

Only after the verification is clean, update copilot/environments/addons/documentdb.yml to the new family, engine version, and parameters, merge it, and run an environment deploy from a clean main:

git status
git log --oneline -- copilot/environments
copilot env deploy --name <env> --diff

An environment deploy applies every pending change under copilot/environments/. The --diff output shows only the addons template URL and none of its contents, so read the git log output to see what else is pending. The deploy takes about 30 seconds and causes no outage. It moves the cluster onto a CloudFormation-owned parameter group, which leaves the instance at pending-reboot.

Reboot the instance in a quiet period, then delete the hand-made group:

aws docdb reboot-db-instance --region us-east-1 --db-instance-identifier <instance id>
aws docdb delete-db-cluster-parameter-group --region us-east-1 --db-cluster-parameter-group-name nexgen-v3-<env>-docdb80-manual

The reboot is a second outage of about 7 seconds. Repeat the verification afterwards, and confirm that planner_version is still 1.0 on the new group. Keep the pre-upgrade snapshot until production is settled.

7. Record the result​

Update this page (the engine version, the parameter group names, and the version history) and the environment reference in Refreshing staging from production. In the nexgen_v3 repo, update CLAUDE.md and the aws and devops skills, and change the clone parameter group in scripts/refresh-staging-from-prod.sh if the staging group was replaced.

Rolling back​

  • Before the upgrade command, nothing has changed. Delete the snapshot and the unused parameter group.
  • After the upgrade, the last resort is aws docdb restore-db-cluster-from-snapshot into a new cluster on the old version, adding an instance, and repointing the services at it. Writes made after the snapshot are lost, and the CloudFormation stack then needs reconciling.

Local development​

The development environment runs MySQL 8.4 and MongoDB 8.0 natively on the Mac, installed with Homebrew, and no longer runs them in Docker containers. MongoDB 8.0 matches the deployed major version. Nexgen V3 local development has the setup steps and the development logins. A local MongoDB does not reproduce DocumentDB's query planner or its aggregation limits, so a query that is fast locally still needs checking on staging. npm run test:perf-staging from the nexgen_v3 repo root runs the DocumentDB performance suite against staging.