Simplifying Configuration Management in Pure Python

Hey! Alejandro here, from Mostly Harmless Ideas. Welcome to my coding blog. If you are reading this article via email and are wondering why are you getting yet another newsletter, that’s because I keep my coding-specific articles, which are way less frequent, in Hashnode rather than Substack. If you are not interested in coding at all, feel free to unsubscribe (at the bottom of this email) from this newsletter, and you’ll keep receiving all my other, non-coding-focused articles from Substack.

In modern software development, managing configurations and feature flags effectively is crucial for maintaining flexibility, ensuring smooth deployments, and enabling rapid iterations. This article presents a straightforward approach to handling feature flags and configurations in a Python project: using environment-specific configuration files and relying on no external dependencies. We will explore the motivation behind this method, provide implementation details, and highlight potential pitfalls to avoid.

Motivation

As applications grow, so do their configurations. Development, staging, and production environments often require distinct settings. If not appropriately managed, managing these configurations can become cumbersome. Traditional approaches often involve plain text files, using nested structures, or external libraries to manage configurations, which can lead to complexity and confusion.

The proposed method simplifies this by utilizing separate Python files for each environment. By dynamically loading the appropriate configuration based on an ENVIRONMENT variable, we can keep our code clean, organized, and easily managed.

An example of a well-established framework that follows this practice is Django. Django projects often separate environment-specific configurations into distinct Python files, such as development.py and production.py, to manage settings for different environments effectively. This approach is widely recommended in the Django community as it simplifies transitions between environments and allows for better organization of configuration settings.

Implementation Details

Step 1: Create Environment-Specific Configuration Files

We will create separate configuration files for each environment (e.g., production.config.py, staging.config.py, development.config.py). Each file will define global variables for feature flags and other configuration options.

Example: production.config.py

# production.config.py 
NEW_FEATURE = False 
EXPERIMENTAL_FEATURE = False 
DATABASE_URL = "https://prod.database.url"`

Example: staging.config.py

# staging.config.py
NEW_FEATURE = True 
EXPERIMENTAL_FEATURE = False 
DATABASE_URL = "https://staging.database.url"`

Example: development.config.py

# development.config.py 
NEW_FEATURE = True 
EXPERIMENTAL_FEATURE = True 
DATABASE_URL = "https://dev.database.url"`

Step 2: Create a Loader Function

Next, we will create a config.py file that contains a function to load the appropriate configuration based on the ENVIRONMENT variable.

# config.py
import os
import importlib

def load_config():
    environment = os.getenv("ENVIRONMENT", "DEVELOPMENT").upper()

    try:
        return importlib.import_module(f"{environment.lower()}.config")
    except ImportError:
        raise Exception(f"Configuration for environment '{environment}' not found.")

Step 3: Access Configuration in Your Application

Finally, we can access the loaded configuration in our application by calling the load_config function.

# main.py
from config import load_config

def main():
    config = load_config()

    if config.NEW_FEATURE:
        print("New Feature is enabled!")
    else:
        print("New Feature is disabled.")

    print(f"Database URL: {config.DATABASE_URL}")

if __name__ == "__main__":
    main()

Step 4: Set the Environment Variable

Before running your application, set the ENVIRONMENT variable in your terminal (or in a .env file if you are already using it):

export ENVIRONMENT=production python main.py
# or staging or development

Advantages and Limitations

When managing application configurations and feature flags, developers often face a choice between using environment-specific Python configuration files and traditional .env files that are most commonplace. Each approach has its own set of advantages and limitations. Here, we will explore these aspects in detail.

Advantages of Configuration Files

1. Version Control: Python configuration files can be checked into version control systems (like Git) without exposing sensitive information. This allows teams to track changes, collaborate effectively, and maintain a history of configuration adjustments across different environments.

2. Computed Values: Python files allow for the inclusion of computed values and logic. This means you can dynamically generate configurations based on other parameters or conditions, providing greater flexibility than static key-value pairs in .env files.

3. Modular Design: Common configuration values can be refactored into separate modules imported by relevant configuration files. This modularity promotes code reuse and reduces redundancy, making maintaining configurations across multiple environments easier.

Limitations of Configuration Files

1. Security Concerns: Unlike .env files, which are often excluded from version control (ensuring sensitive information remains private), Python configuration files may inadvertently expose sensitive data if not managed properly. Developers must exercise caution to avoid committing sensitive information to the repository.

2. Risk of Misconfiguration: The flexibility of Python configuration files can lead to developers unintentionally modifying production settings during development or testing phases, potentially causing disruptions in live environments.

3. Learning Curve: This approach may not be as familiar to new developers as the more widely adopted .env file convention. As a result, onboarding new team members may require additional training or documentation to understand the custom setup.

Advantages of .env Files

• Simplicity: .env files are straightforward text files that store key-value pairs, making them easy to read and edit.

• Environment Separation: They clearly separate settings for different environments without requiring code changes.

• Common Practice: The use of .env files is a well-established practice in many development communities, making it easier for new developers to adapt.

Limitations of .env Files

• No Security by Default: While .env files can be excluded from version control, they are still plain text files that can be easily accessed if not adequately secured.

• Human Error: Developers may also accidentally modify or misconfigure environment variables, leading to potential issues in production.

• Limited Functionality: .env files do not support computed values or complex logic, which can limit their usability in more dynamic applications. However, this may be considered a strength in reducing configuration complexity.

Striking a Sane Balance

There is no free lunch, or so the saying goes. However, you may strike a sane balance in this case by externalizing sensitive and static configuration options to .env files and keeping more dynamic options (e.g., feature flags) in Python files. This way, you maintain flexibility for rapidly changing or very development-specific configuration variables and leave the most sensitive, project-wide configurations for maintainers or DevOps engineers to handle.

Obvious Pitfalls to Avoid

While this approach offers a clean and organized way to manage configurations, there are some pitfalls to be aware of:

1. Security Risks: I already said it, but to reiterate, DO NOT STORE sensitive information (e.g., API keys and database credentials) in plain text within your configuration files. Instead, consider using environment variables or, even better, secret management tools.

2. Version Control: As a follow-up to the above, ensure that sensitive information (e.g., what’s included in .env files) is excluded from version control (e.g., using .gitignore). This helps prevent accidental exposure of secrets.

3. Error Handling: Implement robust error handling when loading configurations. If a specified configuration file does not exist or fails to import, your application should gracefully handle this in upstream code rather than crash unexpectedly.

4. Documentation: Clearly document each configuration file and its purpose. This will help team members understand the differences between environments and make necessary modifications.

5. Testing: Regularly test your application in all environments to ensure that the configurations are correctly applied and that features behave as expected.

Conclusion

Managing feature flags and configurations in Python can be streamlined by using environment-specific configuration files loaded dynamically based on a ENVIRONMENT variable. This approach enhances organization, simplifies edits, and provides clarity when dealing with multiple environments. By being mindful of security risks and other pitfalls, you can implement this strategy effectively in your projects, leading to smoother development cycles and more robust applications.

Choosing between Python configuration files and .env files depends on your project's needs and the team's familiarity with each approach. While Python configuration files offer greater flexibility and modularity, they come with risks that require careful management. On the other hand, .env files provide simplicity and are widely recognized but lack built-in security features and advanced capabilities.

Ultimately, understanding the advantages and limitations of each method will help you make an informed decision that aligns with your development practices and security requirements.

Now, I’m interested in reading your thoughts. Have you ever used Python configuration files? If not, why not? If yes, what was your experience?