By Sarah Ferguson, Sri Paleti, and Ikagarjot Singh Kamra
In today's rapidly evolving tech landscape, it's essential to swiftly deploy new features and functionalities to production. Teams are pushing code at a rapid pace, aiming to collect feedback and iterate quickly. One tool that is revolutionizing software development with its feature flags and targeting capabilities is LaunchDarkly. It enables teams to iterate rapidly, gather feedback and innovate confidently. Let's delve into how this tool is reshaping software deployment practices.
Why use feature flag Toggles?
Feature toggles, also known as feature flags, offer a range of benefits that streamline the development and deployment processes. These include:
- Continuous deployment: Feature toggles enable continuous deployment by allowing new features to be deployed to production without being immediately visible to all users. Developers can push incomplete or experimental features to the live environment, test them in a controlled manner and gather feedback without disrupting the user experience.
- Incremental releases and faster feedback loops: With feature toggles, features can be released incrementally to different user segments, facilitating a gradual rollout that reduces risk by limiting exposure to potential issues. This approach allows for A/B testing to evaluate the impact of new features on user engagement and behavior. Additionally, by deploying features early and iterating based on real user feedback, teams can shorten feedback loops and improve the overall quality of the product.
- Rollbacks and hotfixes: In case a new feature causes issues, feature toggles provide a quick and safe way to disable the feature without needing to redeploy the entire application. This ability to toggle features on and off reduces downtime and enhances system stability, making it easier to manage rollbacks and hotfixes.
- Enhanced testing: By isolating new features behind toggles, teams can perform more rigorous testing. Features can be tested in production-like environments with real user data, allowing for more accurate performance assessments and early detection of bugs. This approach supports both manual testing and automated testing strategies.
- User-specific customization: Feature toggles allow for personalization by enabling or disabling features for specific users or groups. This capability is particularly useful for beta programs, targeted promotions and phased rollouts, providing a tailored user experience and gathering targeted feedback.
- Reduced risk: Deploying large features in a single release can be risky. Feature toggles reduce this risk by allowing features to be developed, tested and deployed incrementally. This approach mitigates the potential for significant failures and facilitates a more controlled and measured deployment process.
For additional feature toggle benefits, read Feature Toggles (aka Feature Flags) by Peter Hodgson.
Lifecycle
Here are the key stages in the life cycle of a feature toggle, ensuring a smooth and controlled rollout.
Figure 1 Staged rollout with feature flags
Evolution of feature toggles in our backend
Our backend is the backbone upon which our offerings are built. Most of the functionality we provide is implemented in or against the backend. Because of this, it’s critical that when we release new features that they are behind feature toggles. We’ve gone through a few different iterations of our feature toggle setup. First, we used Confd. Confd is a properties file base system. Although it worked, it wasn’t as flexible as we wanted and had a lot of code duplication. Our biggest issue with it was if we wanted to update the value of a feature toggle, we would have to update the required properties file and redeploy. We also had multiple files we needed to update in order to add a new feature toggle. Due to this duplication, we often had feature toggles that were lost or forgotten about, which could introduce unneeded complicity and risk into our code base.
Because of this, we used Amazon Web Services Systems Manager (AWS SSM) parameters to update the values of our feature toggles without needing to redeploy our backend. However, the SSM parameters were not decoupled from Confd. On application startup, our code would load the feature toggle keys and values from Confd. Then, if a feature toggle with the same key existed as an SSM parameter, it would use the value from AWS. The negative aspect of using AWS is that we need to create code infrastructure (in other words, even more code) in terraform to create the SSM parameters.
Switching to LaunchDarkly is a huge upgrade from what we had before. Now we don’t have to update as many property files or create SSM parameters. We can change the feature toggles values on the fly. We also don’t have as much coupling with the property files as our previous setup (although we still currently use them for integration tests). One of our upcoming changes for this is to upgrade our integration tests so that they also use the LaunchDarkly framework.
Figure 2 Benevity ecosystem utilizing LaunchDarkly for feature flag management
Relay Proxy for enhanced availability
While LaunchDarkly recommends a Relay Proxy setup for PHP applications, we've successfully extended it to certain Java applications for enhanced availability during LaunchDarkly outages or connection issues encountered in lower environments.
Before configuring the Relay Proxy, it's crucial to carefully consider your desired configuration, plan the architecture, and then deploy it effectively. This ensures optimal performance and avoids potential issues.
You can also decide whether the Relay Proxy is appropriate for your configuration based on the Relay Proxy use cases provided here.
Configuration considerations
LaunchDarkly's default configuration involves its SDKs directly connecting to LaunchDarkly to fetch and update feature flag data. This data is then stored in memory for flag evaluations.
As an alternative, we can leverage a database (e.g., Redis) as the feature store. Here, two configuration options are available:
Database-backed store with LaunchDarkly updates: Substitute the in-memory store with a database while retaining the connection to LaunchDarkly for updates.
Database-only store (Isolated from LaunchDarkly): Utilize the database solely for flag data, without a direct connection to LaunchDarkly.
The first approach, while enabling multiple application instances to share the same database and connect to LaunchDarkly, introduces redundant data writes (harmless but inefficient). To address this redundancy, we can implement a persistent feature store that doesn't require individual instance connections to LaunchDarkly. A daemonized Relay Proxy can then be used to update the database with the latest flag data.
Infrastructure setup:
Persistence store: AWS ElastiCache for Redis is used to store feature flags persistently.
Relay proxy: This runs on an AWS Fargate instance within our cloud environments. It connects to LaunchDarkly's streaming API and proxies that connection to internal clients.
Benefits of Relay Proxy with Persistence
Our dependence on an external service (LaunchDarkly) for flag management introduces potential downtime risks. Additionally, situations can arise where production defaults are set to “off” before code activation, or where inactive LaunchDarkly leads to unintended code deactivation. In such scenarios, the Redis cache acts as a critical failsafe, ensuring feature flag availability even during LaunchDarkly outages.
Additional considerations
- Consider implementing monitoring and alerting for the Relay Proxy and Redis store to ensure their health.
- Regularly update the Relay Proxy and SDKs to benefit from bug fixes and security improvements provided by LaunchDarkly.
Local testing with Redis and LD relay containers Docker containers
For local testing, we use docker containers when introducing new features. In order to test the LD integration with our application, we create local containers for both Redis and LaunchDarkly relay. We inspect the local Redis to confirm the replicated toggle state once the containers are up and running.
For more information on creating a Redis Docker container, go to the official documentation. For the official Docker Hub LaunchDarkly relay image, go to their README.
Conclusion
LaunchDarkly offers a powerful and versatile feature management solution that empowers development teams to streamline software deployment processes. LaunchDarkly simplifies our implementation and management of feature toggles. With the configuration of a relay proxy with a persistent data store, we’ve enhanced our feature toggle availability and resilience.
By adopting LaunchDarkly, development teams can gain a significant edge in today's rapidly evolving software landscape. The ability to iterate quickly, gather real-time feedback, and mitigate risk through controlled deployments paves the way for a more efficient and successful development process.
This article was written in collaboration between:
- Sarah Ferguson: Senior Software Developer
- Sri Paleti: Senior Software Developer
- Ikagarjot Singh Kamra: Staff Developer