Feature flags are a mechanism for toggling application features on and off as apps move between development and production DevOps environments. Until 2020, Climate LLC, previously known as The Climate Corp., had used a homegrown feature flag utility. That year, it signed on with a commercial vendor, LaunchDarkly, to support feature flag use at scale, with role-based access control and finer-grained feature targeting than it could achieve with its in-house tool.
“Over the last several years, we’ve adopted continuous delivery and testing in production,” said Artie Lee, director of engineering at Climate LLC. “Previously, we used QA, staging and other environments, but our developers now commit straight to production after automated testing.”
This approach is called trunk-based development, in which developers commit changes to a core code branch, rather than merging changes from separate long-lived feature branches. For Climate LLC engineering teams, it has improved the speed and safety of feature updates by forcing smaller and more frequent changes. Feature flags, which can be flipped on and off like a light switch, offer near-instant rollback for production apps should they meet with errors, Lee said.
The stakes for production app reliability are high for Climate LLC’s agricultural customers, who often have limited windows of time to plant and harvest crops, he added.
“If something goes sideways, we can roll back quickly to a known good state, rather than redeploying the app,” he said. Redeployment might have to go through Apple and Google’s mobile app vetting process, which can take anywhere from 24 hours to 2 days, far too long for farming customers to wait for access.
Artie LeeDirector of engineering, Climate LLC
“In the agricultural space, tech like this is still nascent, and if your mean time to recover your app impacts your users for two days during a critical time, it will make or break whether they use it again,” he said. “As a farmer, you get about 40 chances to succeed in a lifetime — 40 yearly seasons. As a software engineer, I get 40 chances every week to write code.”
These crucial days each year are also becoming less and less predictable due to climate change, which makes the reliability of crop science mobile apps even more crucial. “If you’re disrupting planting and harvesting, those two days can represent a significant portion of a farmer’s annual income,” he said.
Since 2020, the number of chances for Climate LLC developers to make or break its iOS and Android apps has also increased exponentially, as it scaled from a US-only business to availability in 23 countries and 18 languages, with some 200 million acres under management as of this year.
“We need to make sure the apps are available globally, 24/7, especially with climate change affecting the weather and when planting and harvesting can start,” Lee said. “Those cycles used to be about the same year after year, but they’re not the same anymore.”
Small changes and simple apps key to effective incident response
Another appealing aspect of LaunchDarkly’s commercial feature flags tool is that it can archive old and unused feature flags to avoid clutter in the production environment. Its reporting interface replaced an ad hoc system of snapshots managed by multiple teams using tribal knowledge with the previous homegrown tool, Lee said.
While Climate LLC has about 400 developers, feature flags are primarily used by mobile app development teams, approximately 90 engineers in all, who share a 75-seat license for LaunchDarkly. There have been some lessons learned along the way as developers at the company expanded their use of feature flags, according to Lee, such as the potential pitfalls of too many nested flags.
“If you have a complex feature, you can get into hairy situations trying to test all the different possible code paths,” he said. “We’ve found it’s best to limit the scope of the flag hierarchy to two levels.”
So far, correlating feature flag toggles to issues that may arise in production is still a manual process for Climate LLC’s developers. LaunchDarkly offers automated feature flag triggers that turn flags on and off in response to alerts from observability tools such as Datadog, Dynatrace and Honeycomb, but Lee said he’d also like to start using something like LaunchDarkly’s Jira integration to correlate between production errors and feature flag changes within developers’ existing workflows.
“LaunchDarkly gives us a good audit log of who changed what and why, but [for now] it can be kind of a manual process for engineers to trace an issue back to a specific change,” Lee said.
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.