Empathetic Design – UX for APIs

In a previous blog article API Design, I talk about using a UX mindset when designing APIs.  While, the basic premise of the article is still valid, I think it is more instructive to look a little deeper at the issue.  I had the privilege of talking about this very issue during my developer day keynote at .NEXT in Anaheim last week. 

My journey at Nutanix is just under 8 months old now and I was hired to be an architect for Prism APIs.  One of the key factors in my joining the team was the company’s Design is Everything value.  Having been on a passionate quest to bring usability to enterprise product manageability for a number of years now, this commitment to design (sadly still missing in just about any enterprise product) was extremely exciting.

Not surprisingly, I discovered that the focus on design was restricted to the User Interface (UI), which is the best UI in the world, bringing consumer grade experience to enterprise product manageability.  Unlike other places though, this commitment to design meant that there was a huge opportunity to create something really unique, innovative and differentiating and make API consumption delightful to the customers.

For developers by developers

While no one will disagree that making an API simple to consume is a worthy goal, in the real world, this doesn’t happen very often. There are several reasons for this:

1. The scope of APIs is much larger than a UI that purports to achieve similar goals. APIs are intended for the API consumer to build any arbitrary application using the responses returned by the API (with the UI client being one such instance of a consumer). For this reason, API response payloads tend to be more verbose in order to address a broad spectrum of (mostly unknown) use cases;
2. User Experience (UX) focuses solely on the UI and not on the API. Although most people mistake UX to be just the Visual Design aspect, UX involves interaction design and information architecture as well. However, the constrained environment of a UI, makes the UX problem relatively simple compared to doing UX for APIs;
3. An API consumer is typically a developer who is building an application using the platform serving those APIs, or an administrator looking to automate important aspects of their daily jobs to achieve some enterprise business objective. The API producer is also a developer who designs and develops the APIs for the consumer. Broadly speaking, one could make the case that APIs are produced for developers by developers.

Given this landscape, it is not too surprising that most APIs are complex to use. Engineers love complexity – indeed they wear it as a badge of honor; it is the very reason for their existence, and this is true for both producers and consumers of an API. Consequently, simplification tends not to be treated as a very important consideration for API producers, who tend to err on the side of providing more information to the consumer, for fear of missing out on supporting some unknown use case. It is not uncommon to find that REST APIs are nothing more than glorified database object browsers, with the data model reflecting the structure of the persisted form of the data. This obviously means that an API consumer cannot understand the data models without being exposed to the underlying design and implementation choices made by the producer. What’s worse, it is not uncommon for APIs to reflect the organizational structures that exist at the time that the API is produced.

Not all complexities are equal

Fortunately, we can alleviate this problem by taking a more nuanced view of the producer and consumer developer personas. While on the surface, complexity seems like a common thread tying the two entities, they really are different. For a producer, complexity revolves around making the APIs responsive at scale, striking a balance between too much and too little data in the API response, and the internal services that participate in producing these APIs. On the other hand, complexity from a consumer’s perspective is around solving their IT business problems (which probably involves interacting with multiple API platforms) on behalf of their users and making sure that their applications are responsive at their scale (which is significantly different from the scale considerations of an API producer).

What is therefore required is that the API Producer must consider the design of the API from the perspective of the API Consumer and design the API payloads to make that as easy as possible. Specifically this means:

1. Not creating APIs that are easy to write and develop but that are easy to consume. This necessarily means, that the complexity of the API platform increases significantly – but this is the engineering challenge to solve for the producer. The producer absorbs the complexity so that the consumer doesn’t have to;
2. Follow consistent documented conventions and stick with them across all the APIs. Given all the varied opinions around many REST related topics, this is not a right or wrong approach – it is best to pick a method and stick with it consistently;
3. Have a consistent and scalable mechanism of communicating the state on the server to the client. Making multiple API calls to achieve something is not usually a problem. Not knowing what API to call is more annoying to the consumer. Proper use of hypermedia links in the response payloads go a long way in making APIs more consumable;
4. Consider compiled clients (written in Go/Java etc) along with interpreted clients like Python/JavaScript. This is important because the former tend to break when schemas change more easily than the latter. This is often ignored but can be relatively easily avoided by good and scalable schema designs.

I use the term Empathetic Design to describe what an API producer needs to keep in mind when designing APIs. By understanding of the differences in the definition of complexity and empathizing with a fellow engineer who consumes these APIs, we will create APIs that are delightful to use.

Further exacerbating the usability challenge is the considerable amount of disagreement in the industry around best practices for API development. A simple search on API best practices, on your favorite search engine, will yield multiple conflicting opinions and passionate debate around:

1. API versioning: This is probably the most passionately discussed topic with opinions scattered around the entire spectrum from API endpoints shouldn’t be versioned (Roy Fielding), to headers, URI parameters and everything in between. You will find that each of the major API platforms, like Amazon, Twitter, and Microsoft have a different way of handling this;
2. Error handling: HTTP status codes are woefully inadequate to communicate to the client exactly why something failed. Opinions around how to communicate these errors fluctuate wildly;
3. Backward Compatibility: Most definitions of backward compatibility involve the obvious ones around not tightening constraints, not removing fields from the response etc. This narrow definition of backward compatibility ignores other ways in which compatibility can break even if all the currently identified best practices are followed.

API Best Practices – The Nutanix Way

The Prism UI has shown that consumer grade experiences for enterprise applications is not an oxymoron. We now want to extend that thinking to APIs as well and lead the way in defining a set of best practices around API development that are less pedantic, recognize real world problems in the consumption of APIs, and more prescriptive. Ultimately, our goal is to take a best of breed approach to solving this problem rather than to get hung up on a theoretical definition of a paradigm.

Accordingly, these are the axioms around which Nutanix APIs will be built:

1. APIs will be versioned using a semantic versioning model (major.minor). Regardless of the debate around REST API versioning, real life usage of an API requires that servers serving APIs and the clients consuming them will evolve independently and often without notice. It is critical for the server to know what a client is capable of handling so that an appropriate response can be sent;
2. Within a major version (also called API family), backward compatibility is guaranteed across all minor versions (also called revisions) of that family;
3. Use of the Open API Specification (OAS) to communicate all aspects of an API endpoint and its model;
4. Using a development platform that makes it easy for developers to produce these APIs. An API first development cycle typically involves the following steps:
   1. Define API endpoints/models using the OAS in YAML;
   2. Generate client code and server side interfaces using code generation tools;
   3. Complete the implementation of the API interfaces; and
   4. Package, test and deploy the API.
5. Use the OAS specification as the single source of truth and automate extensively to gain velocity without compromising flexibility.

Defining API endpoints is typically not developer friendly, lacks appropriate tools and uses a process that is quite distinct from software development in other languages. Often developers work around these limitations using custom scripts, making it difficult to manage the evolution of these APIs. Accordingly we are creating an opinionated development platform that will make API development simple and less error prone for the API designer .

In subsequent blog posts, I will cover each of these aspects in more detail. By using empathy as a fundamental value, we want to make both producing and consuming APIs delightful.