This year I was invited to O'Reilly Fluent to present at the awesome O'Reilly Fluent about how crossing the chasm is a difficult transition process for any product or company, and how good documentation can help with that.

You can watch my talk here, or follow along this written version.

In this post I’m going to try to convince you of three things:

• Crossing the chasm is hard and it can drain your engineering resources
• Documentation can help avoid that
• But only if you’re using the right process to create docs

But for that, I need to explain you what this "chasm" thing is all about.

The chasm is a moment in your product lifecycle when you’ve already convinced a lot of early adopters to use your product, and we’re now in the process of convincing everyone else to use it too.

And this is a painful moment for every product and company, no matter how big or small your team is. Of course Docker is no exception.

This transition process is hard because early adopters and mainstream users and fundamentally different. These users have totally different needs and goals.

I mean, if you look at your early adopters, they are mostly hacker and tinkerers at heart. They loooove playings with products that are:

• New
• Shinny
• Almost for sure are going to be buggy

Early adopters have fun understanding what your product is and how it works.

In fact they have so much fun, that sometimes they don’t even care if your product creates more problems than the ones it solves.

Early adopters love tuning and troubleshooting. They take pride in finding and reporting bugs.

If you’re like Docker and happen to have an open-source product, your early adopters will probably submit pull request and help you fix the problems they found themselves.

Early adopters also care a lot about the underlying technology that you’re using in your product and the way your product is architected.

This is the kind of user that prefers that you use a new and exciting technology, even if the technology that you’re using is not backwards compatible, or doesn’t integrate well with other technologies.

And early adopters also prefer technology for cheap or free. Of course there are a lot of exceptions to all this, but since early adopters are playing with all kinds of technologies and products, they don’t want to be paying for a product that they might discard two weeks later.

Now, contrast this with mainstream users.

Mainstream users use products that save them time or money.

Yes of course. Mainstream users also use products like Farmville that costs them time and money, but that’s beside the point.

The point is that it’s hard to convince mainstream users to use your product because they’re very comfortable with the products they’re using. And they’re not looking to replace them.

They also know that the first iterations of your product are going to be inherently buggy, so they’ll wait and see if your product gets more stable.

Since mainstream users are looking for products that can save them time or money, they value products that work out of the box.

They understand that the time they’re spending in integrating technologies, is time they’re not spending delivering functionality to their end users.

Mainstream users might have a lot of applications running and they want to minimize the risks of adopting your product. So they expect that your product works well with whatever systems they already have in place.

They won’t replace all their tech stack by some shiny new technology. They’ll create a roadmap that allows them to slowly refactor their apps and migrate to your product, but only if proves worthy.

And since mainstream users want to save money or time they’ll do a cost/benefit analysis on your product, and they’ll be happy to pay for your product if they think they’re achieving that.

So as you can see early adopters and mainstream users have totally different needs and goals.

And so, this transition from early adopters to mainstream users can be a make it or break it moment for your product or company. And that’s why some people call this transition “the chasm”.

Of course, you can decide not to change anything in your product or team, and try to continue selling to the exact same users you are currently selling to. But there’s only a finite amount of early adopters. So if you decide not to change your strategy your product won’t have global reach.

Ok, so now that you understand what this "chasm" thing is, let me explain the impact that this transition can have for your teams.

And for that, we need to compare the user journey for both kinds of users, since the very beginning. From the point they learn about your product, until the point they are using your product in business critical areas.

Let’s start with early adopters.

An early adopter will probably learn about your product on Hacker News, Product Hunt, or some other place where people gather to discuss unicorns.

If they think your product is cool enough, they’ll give it a try.

• If you’re product is open-source, they’ll fork your repository and build the product for themselves
• If you’re close sourced, they’ll install your binaries

Once they get your product to run, they’ll start using it to hack something together.

And if they like what they’re seeing they’ll continue using it in other projects.

If you’re product product is open-source, they’ll probably start learning about the product internals.

And all this journey, from beginning to end is completely done in self-service mode. They’ll probably read your blog post and API docs, but that’s it. They don’t need to reach out to you, to start using your product and build amazing things.

The other interesting part is that this whole journey takes them days, or weeks at the most.

Now contrast this with mainstream users, specially if they work in the enterprise space.

Things are completely different!

I mean, mainstream users learn about your product from someone else. Almost for sure an early adopter.

So if they believe a product like yours can save them time or money, they’ll start comparing your product with other alternatives.

Even after they think your product is the best one for them.

They won’t believe in everything you say. So they’ll start doing proofs of concepts with your product to test out if your product really has all the functionality that they need.

Of course, proofs of concept can test a few things, but things like non-functional requirements are hard to test. So they’ll be going back to you to ask you about requirements like:

• Performance
• Security
• Scalability
• High-availability

And all the other "ilities".

Since mainstream users want to minimize the risk of adopting your product, they'll start small at first, and then expand their adoption.

Of course, along the way, they’ll be using your product like no one has ever done before. So it’s very possible that they start running into problems.

And once that happens, they’ll want you to be there to help them out as fast as possible.

And as they’re adoption increases, they’ll want to make sure that:

• They’re using the best practices so that they don’t run into problems later on
• Have a way to bring their new team members up to speed using your product

And of course if they really like your product they’ll start demanding new features, and they want to have a way to upgrade to the latest version with no impact to their users.

So as you can see, the user journey for mainstream users is totally different from early adopters. While early adopters are running mostly in self-service mode, things don’t really work that way with mainstream users.

You’ll have to create several points of contact with them, Some points of contact can be mostly self-service, like a marketing website to explain what your product does, but a big part of this journey requires you to hand-hold users along the way.

And this places a big burden on your team. You’ll be having several teams interacting with users, and they’re doing it for long periods of time. Because if you’re lucky this whole user journey from beginning to end can take weeks, but most of the time this journey takes months.

So at this point I bet that you're thinking. This is all good because you work in the engineering organization and most of what I’m talking about is not really related to that.

I mean, I’m mostly talking about marketing, customer success, support, training, and sales, right?

But if you look at the way your team is structured, I bet it looks like this.

If your team looks like this and:

• You’re product is new or moving very fast
• Marketing, services, support, and training teams are also probably new or can’t keep up with the changes you’re making to the product

Guess who will have to help them out once they start having questions? It bet you already know the answer to this one.

The thing is, in the age of open floors and instant communication, the people in all these organizations are going to be communicating in real time.

And while real time communication is awesome because it allows information to be spread faster, and allows people to solve their problems faster, this also means, that it’s very easy to drain your engineering resources with constant interruptions.

I’m not advocating for silos here, but you need to be careful on how you manage your engineering time, because you might be putting the next generation of your product at risk.

And this is especially critical when you’re crossing the chasm. At this stage of your product it’s not really obvious to anyone if your product is the clear winner in your product category. So if you fail to ship, you’re also losing a lot of ground to your competition, that will be hard to regain.

Since this is basically a problem on how to scale communication, both with your external users and your internal teams.

I think that having good product docs, can help you solve this issue.

Good docs can provide a self-service way for your users to solve their own problems. This has 2 side effects:

• It can reduce dramatically the number of interactions that your users need to have with your team,
• It will make the few interactions that they need to have with you, richer.

With good documentation, users can do their homework beforehand and ask you deep and meaningful questions, if they can’t find an answer by themselves.

And the same thing can happen with internal interactions too. Good docs can serve as a buffer between internal teams.

In fact if you’re building your applications with micro-services or any sort of service-oriented architecture, you can use documentation to help other engineers understand how to use and integrate with your services. This way you don’t have to answer to the same question over and over again.

So at this point you might be thinking... Ok, but I use slack, and all my conversations happen in public channels where others can see. So if someone from another team asks me for help, everyone will be able to see my answer and learn from it, right?

In theory yes, but my experience tells me it just never happens. Since Slack allows you to have parallel conversations in real time, what ends happening is that only you and the people following the conversation in real time will benefit from the answer.

An example that illustrates this, is that just last week I saw three people, including myself, going into a Slack channel and ask the exact same question in less than two hours.

So I can only imagine how many times this happens throughout all the Slack channels that we have at Docker, in the span of a week or a month.

And it would also be fair if you told me that your product already has documentation, but it doesn’t seem to be solving any of these problems since you still have to answer questions all day long.

And you’re probably right. I see a lot of companies that have documentation but they still have problems sharing knowledge at scale. But I would argue that documentation is just like code. Just because your product has more lines of code, it doesn’t mean that it has more quality.

The same thing goes for docs. Just because you have a lot of docs, it doesn’t mean that they’re good and allow people to solve their own problems before asking you for help.

So here’s a framework you can use to assess the quality of your docs.

Your docs need to be easy to find, up to date, useful, and engaging. In this order.

Let me expand on this

What do I mean by your docs need to be easy to find?

Well it’s not just that you should be able to find your docs on Google.

It also needs to be faster for your users to find an answer to their own question, than to ask a human. For that, you need to help them in the context they are in.

If they’re not sure how a CLI command works, they should be able to get an answer without having to leave the CLI.

The same thing goes, if they’re using a GUI. They need to be able to find the information they need there.

And of course I should also be able to find an answer if they start a Google search.

All this ensures that both internal and external users can get the help they need faster and easier than having to ask someone else.

Something that you absolutely should not do is lock your docs behind a login page.

If you do that, your users won’t be able to use Google to solve their own problems, so guess who they’re going to call for help?

The same thing goes for PDFs. Even though search engines are able to crawl pdfs, no one links back to pdfs, so your docs are never going to rank high on Google.

Okay. Your docs also need to be updated, and this is pretty much self explanatory.

Outdated docs make your users not trust the documentation because it forces them to debug your product to try to understand what is the intended behavior of a feature.

So if an external user sees outdated docs, they’ll probably call support.

If an internal user sees outdated docs, they’ll probably going to tap you in the shoulder to get help.

Now, while all the quotes I have in this presentation are totally made made up, this one was not.

This is a quote from a real user.

And while we had four or five people working in documentation, this user was still being forced to debug and call support because he had lost trust in the docs.

And this taught me the importance of keeping docs up to date.

I also said that your documentation needs to be useful.

And I know that this is totally vague so let me expand on this.

What I mean is that we need to acknowledge upfront that no one wants to read the docs, or reads the docs for fun.

People read docs to solve a problem they’re having.

So to create useful documentation, you need to help users:

1. Create mental models of the features you make available in your product
2. How those features fit the bigger picture
3. How to actually use those features

You’ll never be able to guess all the time what is the problem your users are having. But you can give them the right mental models and examples they can use and adapt to their own situation.

I know that this is still a bit abstract, so let me give you an example in the context of Docker.

Docker three key primitives:

• Images
• Containers
• Registries

So good documentation will explain you what these primitives are, and how they relate to one another.

But this is only the first part!

For our documentation to be useful, it also needs to teach you how images, containers, and registries fit fit a bigger picture.

And of course, after that, the documentation needs to teach how to create images, containers, and registries, with an example that I can use to then extrapolate to my own project.

And finally, your docs need to be engaging.

If no one wants to read the docs to begin with, and you’re making your docs boring, users will probably prefer to endure the pain of learning by trial and error, than having to read your docs.

So instead of making your docs describe in beautiful and lengthy prose how your product works.

You could show it and let users see for themselves.

Throw in code samples, interactive snippets, diagrams or basically anything that allows users scan the docs with their eyes and jump to the part that is meaningful for them.

Ok. So we've covered a lot of ground so far, so let me recap:

1. Crossing the chasm is hard because it requires you to interact with a different kind of user
2. This can drain your engineering resources
3. Docs can help avoid that

But only if they're good enough, and by that I mean findable, updated, useful, and engaging.

But making documentation findable, up to date, useful, and engaging is hard. And most teams don’t have a lot of resources.

Surprisingly, I don't think you need a lot of resources. You probably just need to use a better process to create docs. With the right process you can do surprising things even when you have few resources.

So let me share the process I think you should be using.

Most of us are using some kind of agile methodology to develop our products.

Usually, a feature starts small, and eventually it requires more and more resources to make the feature come to life. After a certain point the feature gets more stable so you start phasing out and allocate people to work on other projects.

If you zoom in on a particular feature, the development cycle kind of looks like this.

Maybe a product manager interviews users and starts a spec of what the feature should be.

Then the UX team comes along and helps define how the user is going to interact with the product.

And finally developers come and do their magic. Once everyone is finished, you have improved an existing feature or created a new one.

Now. Most teams are not creating docs during this cycle.

I would even argue that most teams are not creating docs during this cycle either.

What ends happening most times is that you develop a couple of features, and then later someone remembers.

We need docs!

So someone goes ahead, bites the bullet, and starts writing documentation for the features that already exist.

Of course this is not the whole story. The whole story looks more like this.

The product never stops evolving. So by the time you get to document a feature, the product has changed a lot.

And with this approach it’s really hard to create docs that are easy to find, are always updated, are useful, and engaging.

I’m saying that it is hard because you’re essentially using agile processes to develop the product, but then you’re using waterfall to create docs. And these two approaches end up clashing.

So when you look at this, there’s only two options.

Option A is that you develop the feature, wait for the feature to be done, create docs, and you wait for docs to be ready to release the product.

But this is basically waterfall and we’re in the 21st century so we’re done with that.

Option B is to create docs at the same time as you develop the product.

And this doesn’t make any sense either. Does it? I mean, how can you create docs if you don’t even have a working product?

But counter intuitively, I think this is exactly the process you should be using.

You should be documenting the product while you’re creating the product. And for you to understand why, I think we need to go back in time.

Not so long ago we also thought that first you created the product, and then you would make it more beautiful.

And it took us a while to understand this, but eventually we realized that what we wanted wasn’t so much to create a beautiful product, but to create a simple and easy to use product that happens to be beautiful.

So we’ve realized that you can’t build a product, and then make it simple. Once complexity creeps in, it’s impossible to remove it from your product. Making the product simple, is something that you need to consider from day one.

And it’s kind of the same thing for docs.

The goal is not to check an item on a checklist. The goal is to help both internal and external users to lean your product in a way that scales, so that you have time to work on what matters.

The next generation of your product.

So we shouldn’t fixate too much on whether or not the product is finished.

You don’t need to spend a lot of time on this. The goal is to iterate as you go.

If you think about docs from day one, it means that you need to be on top of the specs product managers are creating. You need to play with the mocks the UX team is creating, and then you can start creating some lightweight docs.

As the feature evolves, you continue to iterate the docs, so that once the product is finished, the only thing you need to do is a small last review.

This also means that if you’re developing a feature that needs more docs you can bring people incrementally to the project.

They’ll ramp up fast, since they can use the docs you’ve been creating.

And I know that this process might seem crazy at first. But I’ve been using it at Docker and other companies before Docker. And I think it really works.

I’ve also noticed that there’s three interesting side effects of using this process.

You have docs in time for a beta. Most teams struggle to create docs in time once someone decides that you’ll be doing a public beta But with this process you’re iteratively refining the docs. So if someone decides to do a beta, you might not have perfect docs, but you’ll have something that everyone can start using.

In fact, I’ve done more than 10 major releases using this process, both at Docker and other companies, and I don’t remember the last time my teams didn’t had docs ready for the beta.

Another side effect of using this process is that when you’re creating docs you’re using the product from end to end.

And this is great because sometimes different development teams are building features without having a lot of visibility on what the other teams are working on. This means that sometimes two teams are creating features that don’t play well with one another.

So when you’re creating docs and using the product from end to end, you’ll realize early on that this is happening, and you still have the time to course correct.

And finally, another side effect of doing docs while you develop is that you’re playing with mocks, and also building, deploying, and using the product as it is evolving.

So you’re essentially an alpha tester and can file bug reports when something is broken and heck if the product is really easy to use or not.

This is feedback that you’ll also get early enough and will have time to fix it.

At Docker we’re following this process and creating docs at the same time as we develop the product.

So let me tell you some of the strategies and tools we’ve been using together with this process to make our lives easier.

This one is probably obvious. Since you’ll be doing specs, UX, development, and docs, having a multi-disciplinary team can really make this process go smoother. It makes communication faster and ensures that everyone is responsible for the end result.

So on paper our teams at Docker look like this.

But in reality, our day to day operations look more like this. We embed product managers, designers, developers, and docs in the same team.

And with this we get the best of both worlds. We ensure that teams are cohesive and can move fast but still maintain horizontal teams that can help cross-pollinate information across the company.

And Docker is still a small company. So sometimes you’ll have the same designer or docs person embedded in multiple teams.

In my own case I’m embedded in three different teams, which is awesome.

It forces me to have a high level picture of how our products are evolving as a whole, and we can work together when I notice that two features don’t play well with one another.

Another thing that we do is continuous integration and delivery, but for docs. This allows us to move faster, because we don’t have to spend time integrating and releasing docs. This also forces us to work iteratively to refine the docs. You write, you merge, and you repeat.

But the docs are live all the time. So other team members can use what you’ve created.

Another thing that you should be doing is to manage docs and code in the same repository.

This way, if your pull request changes functionality, you’re also responsible for updating the docs in the same PR.

We’ve stopped doing this a while ago and we’re now managing docs their own repository. We’re doing this because it was getting hard to maintain docs that look consistent across multiple products.

But if you’re just getting started I recommend that you manage docs and code in the same repository. If later your users start complaining about consistency you can find a way to deal with that.

In terms of technology we use:

• Jekyll, a static site generator
• We write docs in markdown using any text editor
• And use the typical Github pull request workflow

To contribute to the docs, you fork the docs repository, make changes in a branch, and then open a new pull request.

Since we want our community to help us keep the docs updated, we also try to make it easy for anyone to edit the docs. Even if they don’t know Git.

So on every docs page we have a link that allows you to edit the page from the Github UI and start the pull request from your browser.

So you can edit our docs without having to clone the repo into your machine or even open a text editor.

Once you create the pull request, we’ll run some tests, and go through the review process.

When your change to the docs is merged, that triggers a build process on Docker cloud. So Docker Cloud will start building a new Docker image containing all the documentation with the changes you’ve made and everything else needed to run the docs site.

So in our case, our Docker image contains Nginx that we use as a web server. Then we take all the markdown files, run them through Jekyll to create the HTML, Javascript and CSS for the whole docs side, and then we put all those files into the Nginx directory.

And this is the cool thing about Docker. You can package your app with all the dependencies it needs to run.

Once Docker Cloud finishes the build process, it tags that the resulting Docker image as “latest”.

Since the image is tagged as latest, this automatically triggers a deployment process in Docker Cloud.

Docker cloud automatically starts a rolling upgrade of all the docs containers.

Once this is finished all the containers will be serving the documentation with the changes you’ve made.

And since we want our docs to be highly available we run multiple docs containers in different servers. We also run a separate load balancer container to load balance requests and also do TLS termination so that you see our docs site using HTTPS.

And finally something that really helps us is that we try to make communication open by default.

And you don’t need to be an open-source company for that.

Even if your product is close-soured, you can still open source your docs and have your community help you maintain them.

Instead of filing issues with your support team, both your internal teams and power users can help you keep the docs updated.