Before the web, constraints of retail software required rigid, year long schedules. After the web, you could ship code with a few keystrokes. Today, big sites ship several times a day. Then apps made a comeback through mobile, and now we face a clash in process. Web companies want to “move fast and break things,” but apps need more planning and rigor.

When asked how I’d make app development as agile as the web, I say it’s fool’s errand. The App Store is something between web and retail. While you aren’t shipping disks, there are still distribution bottlenecks. These destroy the user-feedback loop that makes agile so useful.

To make matters worse, native apps provide a superior user experience at the cost of greater complexity and fragility.

Bugs Without Mercy

If you ship multiple updates to your site every day, bugs slip through. Damage is often contained to a single page. Thanks to graceful degradation, even if JavaScript breaks, the page is still usable.1 In an app, everything exists in a single process, and one bad “page” crashes everything.

It’s a matter of philosophy: the web forgives. Write something that vaguely resembles HTML if you squint, and the browser finds a way to render it. Apps take a fail fast approach, responding to mistakes with a crash. Partly because it’s easier to debug, and leads to better software. Largely because making the programmer’s life easier has performance implications.

iOS cut us some slack in the last few years. You once had to manually manage memory, and if you got it wrong, you crashed. When Apple introduced ARC, they traded crashes for memory leaks.2

Maybe iOS could catch crashes on a per-screen basis. If there’s a crash in your messaging screen, send them back in navigation. This still wouldn’t solve bugs of correctness, like loading incorrect data in your database, forcing a crash on launch.

Distribution

Websites ship embarrassing bugs all the time. You get away with it because you don’t ship to all users. You roll-out to 1% of users and watch the graphs. If things look good, slowly dial up to 100%.

App distribution is long and painful. App Store review takes days. Once you’re approved, it can take weeks to reach over 90% of your user base, even with automatic updates.

You can’t throttle app updates, but you can copy the web’s strategy of feature-switching. Ship updates with the old and new code wrapped in a switch, and throttle the switch server side.

For example:

if featureManager.hasEnabled("DISPLAY_FANCY_MESSENGER") {
  viewController = FancyMessengerViewController()
} else {
  viewController = OldMessengerViewController()
}

Where does that flag come from? Latency makes it impossible to fetch the value on-demand. Instead, you periodically download a configuration file:

{
  "DISPLAY_FANCY_MESSENGER":false,
  "DISPLAY_NEW_PROFILES":true
}
 

It sounds simple enough, but the devil is in the details.

You can’t fetch and load this config on every launch. A web page waits until all dependencies are met before displaying the UI. On mobile, the network request will take seconds, so you can’t just block the UI.

You’ve got a race condition: by the time the configuration file loads, enabling the new UI, the user may already be in the old UI. Do you suddenly punt them to the new UI? What if they’ve started composing a message? Will you write a migrator? Every corner case requires another feature-switch conditional, increasing cyclomatic complexity.

A safe, simple approach is to wait until the next cold launch to actually load the config file. However, cold launches on iOS are non-deterministic; if the user regularly uses your app, it can go days before relaunching. If a major bug slips through, you can’t wait days to roll back.

State is Hard

The very definition of HTTP is “stateless.” The cloud is the truth, and all required data is downloaded when you visit the site. This dumb, thin client approach works great in a broadband world.

Mobile connections are high latency, low bandwidth, and unreliable. You can’t just show a spinner while the posts in your newsfeed load. In addition, apps maintain the illusion that they’re always active, and never launch or quit. When you leave an app, you should transparently save the user’s state, down to their scroll position.

Maybe this will be solved with hardware. When networks can fetch data in under 40 milliseconds, we can treat mobile apps as a glorified remote desktop, but looking at the LTE adoption rate, this is many years away.

Until then, apps require rich local side storage. With it comes the complexity of DB schemas, data migrations, multithreading, and more. It’s not that hard to get things in a very bad state.

I tell everyone to write a “storage kill switch” in their API client. Like the feature switch configuration, the client regularly polls a file with a version number.

{
  "storage_reset":1
}

The first time the client loads it, it saves that value. If that reset value is ever incremented, the client dumps absolutely everything in local storage, except for authentication credentials. This is a last resort, but it’s better than the alternative of waiting for the App Store to approve an emergency update.

The Feedback Loop

At best, we’ve built infrastructure to mitigate damage. We still lack the feedback loop that makes agile useful. In a perfect world you:

  1. Write code.
  2. Run tests.
  3. Looks good? Merge. No? Go to step 1.
  4. Deploy to 1% of users
  5. Looks good? Go to 100%. No? Go to step 1.

On the web, you can run through this cycle in hours. In an native app, getting from step 3 to step 4 takes weeks.

Automated tests, the golden calf of agile development, are not enough. I think automated testing accelerates development. But I haven’t seen a direct correlation between automated testing and quality. On projects with massive, high quality test coverage, I’ve seen just as many bugs slip through as projects with zero coverage.

If you want to release quality software, you need a culture of quality. There’s a bizarre meme among agile people that formal QA is bad. “QA is everyone’s job,” they say with the conviction of a vaccination skeptic. Yes, but it needs to be someone’s full time job.

Sit down with a software engineer from anywhere but the web, and ask them about QA. Tell a game developer you don’t need it, they’ll tell you you’re nuts. Maybe these agile people have been burned by bad QA, but a great QA team is far from a bunch of monkeys clicking buttons all day.

Formal QA provides a counterpoint to “move fast and break things.” Their job isn’t to say, “No.” It’s perfectly fine to ship bugs– otherwise software would never ship. You need someone who is aware of all the bugs, and help you make the decision if the risk is worth it.

In a web world, companies substitute QA with 1% rollouts, because their users are QA. Even if you set aside the negative user experience for “only 1%”, this is no longer effective as the gap in the feedback loop grows. You may think QA adds friction, but that’s negligible compared to distribution time.

Agile Theater

Grafted to mobile, many agile processes only provide the illusion of productivity.

Take the unicorn of “two week release cycles.” As you build infrastructure for faster releases, simple code becomes unwieldy. Tasks that should take hours take weeks. While your version number grows faster, value delivered to your users drops through the floor.

Imagine you drop a feather off a building, it will accelerate to the ground until it reaches a certain point, its terminal velocity. Its weight can’t overcome drag, so it can’t go any faster.

If you really want the feather to go faster, strap it to a gold bar. It’s expensive, and ruins the feather, but it works.

The physics of an app are a little different. The formula involves the time required to do adequate QA, the consequenes of getting it wrong, and the value of getting features out sooner rather than later. When you reach an app’s terminal velocity, you sacrifice quality. The cost of going faster, without a hit in quality, grows exponentially.

We resist longer schedules because disciplined development is a lost art. At the mere mention of a schedule, people scream, “waterfall!” They think it’s impossible to innovate on a two month schedule, but there’s plenty of apps that do it quite well.

I guess we could blame managers, but that feels like the easy way out. Maybe we can fix things.

When you tell a designer they’ll have to wait weeks to see their work live, they feel stressed. You really don’t know how a design will work until you feel it in your hands. If you think, “That’s your problem,” Of course they’ll flock to the web model. Whether or not it fits, feedback leads to a better design. Scaling is your problem.

Instead, come to designers with solutions. Device feedback can be solved with mobile prototyping tools. In fact, feedback arrives faster than waiting for an engineer to build it. Problem solved.

Engineers fall into a trap where they think the technically superior solution always prevails. Meritocracies don’t exist. The web development model has fifteen years of books, conferences, and marketing behind it. If you want to enact change, prepare to evangelize.

1. If something goes wrong in a fat javascript app like a Google Docs, people know to hit the refresh button to fix things.

2. They also lowered the bar of entry. Now when I interview iOS developers, I always test them on memory management.