How We Migrated Macro to Doppler
and Made Configuration Safer

As Macro has grown, so has the number of services we run to deliver the product experience our users expect.

One recurring headache has been keeping each service's configuration requirements synchronized with the environment variables provided to its container at runtime.

Environment desync

A very common pattern, and one that we originally adopted, is to use infrastructure as code to orchestrate all the environment variable values required by our services. We followed this pattern using Pulumi and TypeScript.

It became quite easy to add a new required config value to a Rust service and forget to update the Pulumi infra to provide it. A typical flow: deploy to development, learn that the deployment had failed, and dig through the logs to identify the missing environment variable. The same failure could then happen again in production if the corresponding configuration change was not applied there.

This caused considerable frustration among our engineers and became a frequent topic of discussion within the organization. I shared that frustration, so I decided to address the problem.

Introducing Doppler

I had used Doppler for several personal projects and appreciated how easy it was to use locally. Its Config Sync feature also made it straightforward to provide configuration values to deployed services.

Alongside the frustration about infra/service environment mismatch, there was also growing frustration over the difficulty of setting up local environments with all the necessary configuration values. Doppler offered a solution to both problems: personal configs allow per-user customisation of specific fields without overriding the values other engineers use.

With Doppler in place, we store each service's configuration values in a Doppler project managed through Pulumi. That gives us one source of truth for both deployed services and local development, making configuration easier to synchronize and update.

Runtime configuration

In this system, configuration refers to the complete set of values a service needs. Secrets — API keys, database credentials — are the sensitive subset of that configuration, while values such as ports and queue limits are not necessarily secret. Environment variables are the mechanism we use to deliver those values to a running service.

For deployment, Doppler Config Sync writes the complete configuration to a secret in AWS Secrets Manager, even though not every value it contains is sensitive. At deployment time, that secret is injected into the service container as a single APP_SECRETS_JSON environment variable containing a JSON object.

This was incompatible with our existing model, which read each value individually using std::env::var, so we adapted macro_env_var and macro_config to support the new runtime flow.

The macro_env_var crate

The macro_env_var crate provides type-safe access to environment variables. After some refactoring and the addition of custom Clippy rules, it became the only approved way to access environment variables in our Rust services.

The updated crate looks for a value in APP_SECRETS_JSON first and falls back to the corresponding environment variable when the key is absent. This lets developers use Doppler to inject ordinary environment variables locally, while deployed services consume the JSON object produced by Config Sync.

The macro_config crate

While macro_env_var provides type-safe access to individual values, the macro_config crate brings those values together into a single typed service configuration. When ConfigLoader loads a type that derives macro_config::MacroConfig, the generated implementation attempts to construct every field in that configuration.

Required values, optional values, and defaults are all expressed in the configuration's type definition, giving the service one schema that can be used both at runtime and by our Doppler validation in CI.

For example, the following configuration uses types created by env_vars! and maybe_env_vars! alongside shared environment-variable types from other crates. Fields such as port and environment define defaults, while the remaining values are validated as the configuration is loaded.

use anyhow::Context;
use database_env_vars::{DatabaseUrl, RedisUri};
pub use macro_env::Environment;
use macro_env_var::{env_vars, maybe_env_vars};
use macro_middleware::auth::internal_access::InternalApiKey;

env_vars! {
    pub struct BaseUrl;
}

maybe_env_vars! {
    pub struct ContactsQueueMaxMessages;
    pub struct ContactsQueueWaitTimeSeconds;
}

#[derive(macro_config::MacroConfig)]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub struct Config {
    /// The services base url
    pub base_url: BaseUrl,
    /// port number of service
    #[macro_config_default(8080)]
    pub port: usize,
    /// The environment we are in
    #[macro_config_default(Environment::new_or_prod())]
    pub environment: Environment,
    /// The connection URL for the Postgres database this application should use.
    pub database_url: DatabaseUrl,
    /// The Redis URI for rate limiting.
    pub redis_uri: RedisUri,
    /// The notification queue max messages per poll
    pub contacts_queue_max_messages: ContactsQueueMaxMessages,
    /// The notification queue wait time seconds
    pub contacts_queue_wait_time_seconds: ContactsQueueWaitTimeSeconds,
    /// The internal api key value.
    pub internal_api_key: InternalApiKey,
}

impl Config {
    pub fn from_env() -> anyhow::Result<Self> {
        macro_config::ConfigLoader::load::<Config>()
            .context("failed to load contacts service config")
    }

    #[cfg(test)]
    pub fn new_testing() -> Self {
        Config {
            base_url: BaseUrl::Comptime(""),
            port: 0,
            environment: Environment::Local,
            database_url: DatabaseUrl::Comptime(""),
            redis_uri: RedisUri::Comptime(""),
            contacts_queue_max_messages: ContactsQueueMaxMessages::new_unset(),
            contacts_queue_wait_time_seconds: ContactsQueueWaitTimeSeconds::new_unset(),
            internal_api_key: InternalApiKey::Comptime(""),
        }
    }
}

Calling Config::from_env() gives each service a single startup path for loading its configuration. If macro_config cannot derive a required field because its value is missing or invalid, the Config value is not created. Instead the loader returns an error and the service fails during startup rather than running with incomplete configuration.

Validating configuration in CI

With runtime configuration loading in place, we still needed a convenient way to test every environment against the same configuration schema before deployment. That led to doppler-config-rs.

I originally created this crate to provide runtime secrets to services. After we adopted Doppler Config Sync that functionality was no longer necessary, so the crate is now used exclusively to validate configuration in CI.

It is straightforward to use: provide a struct that implements serde::Deserialize, a Doppler project name, and a configuration slug. The crate then loads the Doppler values and deserializes them into that struct. The following example shows how we validate a service's development and production configurations.

use macro_env::Environment;

mod config;

const DOPPLER_PROJECT: &str = "<project_name>";

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let dev = doppler_config::DopplerConfig::builder()
        .token_from_env("DOPPLER_TOKEN")
        .config(Environment::Develop.to_doppler_slug())
        .project(DOPPLER_PROJECT)
        .build()
        .expect("valid Doppler configuration");

    dev.load::<config::Config>().await?;

    let prd = doppler_config::DopplerConfig::builder()
        .token_from_env("DOPPLER_TOKEN")
        .config(Environment::Production.to_doppler_slug())
        .project(DOPPLER_PROJECT)
        .build()
        .expect("valid Doppler configuration");

    prd.load::<config::Config>().await?;

    Ok(())
}

By loading both environments into the service's Config type, CI catches missing values and deserialization errors before the service is deployed.

End-to-end configuration flow

The complete flow is:

1. Pulumi manages the Doppler project that contains a service's configuration values.
2. Doppler Config Sync writes those values to a secret in AWS Secrets Manager.
3. At deployment time, the secret is injected into the service container as the APP_SECRETS_JSON environment variable.
4. When the service starts, Config::from_env() invokes ConfigLoader, which uses the generated MacroConfig implementation to construct every field in Config.
5. For fields backed by macro_env_var, the crate reads the corresponding value from APP_SECRETS_JSON. If the key is not present there, it falls back to the individual environment variable used during local development.
6. Defaults and optional values are handled according to the Config definition. If any required value is missing or invalid, macro_config cannot create the Config, and service startup fails with an error.
7. In CI, doppler-config-rs loads the development and production values from Doppler into the same Config type, catching missing or invalid configuration before deployment.

Conclusion

Yes, we could have built this system with SOPS, AWS SSM, or several other providers, each with its own tradeoffs. We chose Doppler because of its easy personal configuration setup and our team's prior familiarity with it.

In practice, missing or malformed required values now fail CI when the Doppler configuration is loaded into the service's typed Config, rather than first appearing as failed deployments. That gives our engineers a clearer local setup and earlier feedback on configuration changes, without inventing a separate validation schema.