Skip to main content

How the dashboard loads its data

The internal dashboard at dashboard.axxya.com is a Next.js frontend and a Flask API in the private axxya-dashboard repository, running as one ECS task (see Deploying the dashboard). It has no database of its own. Every chart is a report built from the APIs of the systems that hold the data, and the dashboard keeps a stored copy of each report so that a page opens at once even though the dashboard is used rarely and the reports take a long time to build.

Data sources​

Each page reads from one or more outside services through the Flask API. The credentials are SSM secrets named in the Copilot manifest.

  • Sales and Finance: Chargebee (subscriptions, customers, invoices, products) and Freshsales (trial contacts and the trial type and module fields). The sales charts share three full Chargebee crawls, which take about 30 seconds together and are the slowest thing the dashboard does.
  • Marketing: Google Analytics 4 (page views, organic sessions, landing pages, the pinned trial pages) and Google Search Console (clicks, impressions, queries).
  • Ads: the Google Ads API for the Axxya Systems account (spend, clicks, conversions, campaigns); see Google Ads API access.
  • Support: Freshdesk (open and recent tickets) and the nexgen reporting API on the production and trial deployments (recent logins).
  • Dev: GitHub Actions runs and pull requests for the nexgen repositories.

Staff accounts are removed from the customer and revenue charts by name, email domain and address lists in the backend configuration.

What happens on a request​

A chart asks the API for one report, for example /api/sales/total?range=six_months. The API answers in one of three ways, in this order:

  1. From memory. The Flask process keeps a bounded in-memory cache of every report and of the source crawls behind them. A report stays fresh for six hours (one hour for Freshdesk and the login lists, one minute for GitHub) and can still be served for a grace period while a background rebuild replaces it. If this process has already produced a fresh result for that exact path and query, the view answers from memory.
  2. From the store. Otherwise, if a stored snapshot of that report exists, it is returned immediately, flagged as stale, and one live rebuild of that report is queued in the background. The chart shows the stored numbers with "Showing previous results; checking for updates" and reads again a few seconds later, by which time the live result is normally in memory.
  3. Built inline. Only when nothing is stored does the request wait for the report to be built. This happens for a report nobody has asked for before, and for everything after the very first deploy into an empty bucket.

Every report response carries cache metadata: updated_at (when the data was last built), fresh_until, expires_at, stale, refresh_failed, and snapshot: true when the answer came from the store. The chart footer shows the updated_at time, so a stored answer is always labelled with the time its data was really built, not the time it was served. Only reports that return without errors are ever stored, so a failing source cannot overwrite good data.

The report store​

Stored reports live in the private S3 bucket nexgen-v3-dashboard-store in the nexgen-v3 AWS account (202077789656, us-east-1). The bucket is created by hand, like nexgen-v3-backups, and the task role gets access to it through copilot/dashboard/overrides/cfn.patches.yml in the dashboard repository. The bucket name is set as DASHBOARD_STORE_BUCKET in the gitignored copilot/dashboard/dashboard.env.

The bucket holds three kinds of object:

  • reports/<path>/<sorted query>.json, one per report, for example reports/sales/total/range=six_months.json or reports/sales/products/default.json. Each holds the report's JSON payload, the cache metadata it carried, and the time it was stored.
  • catalogue.json, the list of reports the background refresh keeps current. It starts from a seed list in backend/services/reports.py (the query each chart issues as mounted, about 35 reports) and learns any other path and query that a user requests, for example a year selection or a five year range. A learned report is dropped after 30 days without a request.
  • status.json, the outcome of the last background run: when it started and finished, how many reports it built, and which ones failed.

Nothing in the bucket is a source of truth. It only holds numbers the APIs can rebuild, and a lifecycle rule deletes report objects that have not been rewritten for 30 days.

Locally, with no bucket configured, the same objects go to the .dashboard-store/ directory in the repository root, so development and the unit tests need no AWS access.

The refresh cycle​

The Flask process runs one background thread, report-refresh:

  • At startup it loads the catalogue and the last status from the bucket, then rebuilds every catalogued report. The in-memory cache is empty after a deploy, so until this run finishes, requests are answered from the stored snapshots.
  • Every six hours (DASHBOARD_REFRESH_INTERVAL_SECONDS, default 21600) it runs again. A run first warms the three shared Chargebee crawls, then builds each report in turn, forcing every source to be fetched once for the whole batch so that reports sharing a crawl do not fetch it twice. Each successful report replaces its object in the bucket, and status.json is written at the end.
  • Support data (Freshdesk and the login lists) is also refreshed when a user looks at it between runs, because its in-memory freshness is one hour: a request past that hour gets the previous result at once while the API is read again.

A failed report in a run leaves its previous object in place and is listed in status.json. The next run tries it again.

Because the refresh thread and the in-memory cache live in the Flask process, the service runs exactly one task with one gunicorn worker. A second task would crawl the APIs twice and race on the bucket.

Refreshing by hand​

There are two buttons, and they do different things:

  • Refresh charts (top right, every page) rebuilds only the charts currently on screen, waits for them, and stores the results. Use it when one page looks out of date. It can take 30 seconds or more when the Chargebee crawls are cold.
  • Refresh all data (Settings > Data) wakes the background thread for a full run and returns at once. The header status changes to "Refreshing data…" and, when the run finishes, to "Data refreshed" with the time; charts that are open re-read themselves. If a run is already in progress the button reports that instead of starting another.

The Settings > Data card also shows where the store is, how many reports are in the catalogue, the last run (time, reason, number of reports, duration, failures) and when the next scheduled run is due.

Checking the store from a terminal​

These are read-only. Use the nexgen-v3 AWS account.

aws s3 ls s3://nexgen-v3-dashboard-store/ --recursive --human-readable
aws s3 cp s3://nexgen-v3-dashboard-store/status.json -
aws s3 cp s3://nexgen-v3-dashboard-store/catalogue.json -

The service logs record the same cycle:

copilot svc logs --name dashboard --env prod --since 6h | grep -E "report store|report refresh"

Expect report store backend=s3 location=nexgen-v3-dashboard-store catalogue=N once at startup, then report refresh started reason=startup|schedule|manual reports=N and report refresh finished ... failed=K duration_s=... for each run, with one report refresh failed path=... error=... line per report that failed. Request lines carry snapshot=1 when the answer came from the store.

Troubleshooting​

  • A chart's "Updated" time is hours old. The time is the data's real build time, so this is expected right after a deploy or an idle stretch until the background rebuild lands, usually within a minute. If it stays old, check the header status for a failure count and the logs for report refresh failed lines naming that report; the usual causes are an expired API key (a 401 from the source) or a Chargebee rate limit. The stored snapshot keeps the last good numbers meanwhile.
  • The header shows "Data refreshed" with "(N failed)". The last run had N reports that returned an error. aws s3 cp s3://nexgen-v3-dashboard-store/status.json - lists them, and the logs give the exception type. Fix the source credential or wait for the next run if it was a transient error.
  • The header shows "No background refresh yet". The thread has not completed a run since the task started and nothing was stored before. Wait for report refresh finished in the logs.
  • The header shows "Data status unavailable". The browser could not read /api/admin/store-status, which usually means the Cloudflare Access session expired. Reload and sign in again.
  • report store backend=file in the production logs. DASHBOARD_STORE_BUCKET is missing from dashboard.env, so the task is writing inside its own container and loses everything on the next deploy. Add it and redeploy.
  • AccessDenied on report store read failed or report snapshot write failed. The task role cannot reach the bucket: the override in copilot/dashboard/overrides/cfn.patches.yml was not applied or names a different bucket than DASHBOARD_STORE_BUCKET. Deploy again from the dashboard repository root so Copilot applies the override.
  • Every chart on a page is slow right after a deploy. The bucket is empty (first deploy, or its objects expired after a month without the dashboard running) so the reports are being built inline. This clears once the startup run finishes.
  • "Refresh all data" says the background refresh is not running. The thread is disabled (DASHBOARD_SCHEDULER=off or an interval of 0 in dashboard.env), which is only meant for local development. Remove the setting and redeploy.
  • Deploying the dashboard, including creating the bucket during first-time setup.
  • README.md in the dashboard repository, "Report store and background refresh", for the code-level description and the unit tests.