OpenStack Common Culture

We are 5 years into the OpenStack ride (and 3 years into the OpenStack Foundation ride), and the challenges for our community are evolving. In this article I want to talk about what I consider the most significant threat for our open source community today: the loss of our common culture.

Over the past year we evolved the OpenStack project model to adopt an inclusive approach. Project teams which work on deliverables that help us achieve the OpenStack Mission, and which follow our development and community practices should generally be accepted under the "big tent". As we explained in this presentation in Vancouver, we moved from asking "is this OpenStack ?" to asking "are you OpenStack ?".

What does it mean to be OpenStack ? We wrote down a set of principles, based on the original four opens that were defined at the very beginning of this journey. But "being OpenStack" goes beyond that. It is to be aligned on a common goal, be part of the same effort, be the same tribe. OpenStack relies on a number of individuals working cross-project (on infrastructure, QA, documentation, release processes, interoperability, user experience, API guidelines, vulnerability management, election organization...). It is because we belong to the same tribe that some people and organizations care enough about "OpenStack" as a whole to dedicate time to those essential cross-project efforts.

This is why we standardize on logged IRC channels as a communication medium, why we ask that every project change goes through Gerrit, and why we should very conservatively add new programming languages to the mix. Some people advocate letting OpenStack project teams pick whatever language they want, or letting them meet on that new trendy videoconferencing app, or letting them track bugs on separate JIRA instances. More freedom sounds good at first glance, but doing so would further fragment our community into specific silos that all behave differently. Doing so would make recruiting for those essential cross-project efforts even harder than it is today, while at the same time making the work of those cross-project efforts significantly more complex. Doing so would make our community crumble under its own weight.

We started this journey with a pretty strong common culture. It was mostly oral tradition. We assumed that as OpenStack grew, our culture would naturally be assimilated by new members. And it did, for quite some time. But today we are at a point where we dramatically expanded our community (we doubled the number of project teams over the last year) and our common culture did not naturally transmit to newcomers. Silos with local traditions have formed. Teams don't all behave in the same way anymore. Most team members only care about a single project team. We struggle to move from one project to another. We struggle to provide common solutions that work for everyone. We struggle to recruit for cross-project efforts more than we ever did. OpenStack's future as a community is at risk. It's time to hold the culture line, rather than time to further relax it.

It is also more than time that we document our common culture, so that it can be explicitly communicated to everyone in the OpenStack ecosystem (current and prospective members). We started a workgroup at the Technical Committee, held a virtual sprint to get a base version written, and now here it is: the first version of the OpenStack Project team guide. Read it, refer to it, communicate it to your OpenStack community fellows, propose changes to it. It is an essential tool for us to overcome this new challenge. It's certainly not the only tool, and I hope we'll be able to dedicate a cross-project session at the Mitaka Design Summit in Tokyo to further discuss this topic.

The Age of Foundations

At OSCON last week, Google announced the creation around Kubernetes of the Cloud-Native Computing Foundation. The next day, Jim Zemlin dedicated his keynote to the (recently-renamed) Open Container Initiative, confirming the Linux Foundation's recent shift towards providing Foundations-as-a-Service. Foundations ended up being the talk of the show, with some questioning the need for Foundations for everything, and others discussing the rise of Foundations as tactical weapons.

Back to the basics

The main goal of open source foundations is to provide a neutral, level and open collaboration ground around one or several open source projects. That is what we call the upstream support goal. Projects are initially created by individuals or companies that own the original trademark and have power to change the governance model. That creates a tilted playing field: not all players are equal, and some of them can even change the rules in the middle of the game. As projects become more popular, that initial parentage becomes a blocker for other contributors or companies to participate. If your goal is to maximize adoption, contribution and mindshare, transferring the ownership of the project and its governance to a more neutral body is the natural next step. It removes barriers to contribution and truly enables open innovation.

Now, those foundations need basic funding, and a common way to achieve that is to accept corporate members. That leads to the secondary goal of open source foundations: serve as a marketing and business development engine for companies around a common goal. That is what we call the downstream support goal. Foundations work to build and promote a sane ecosystem around the open source project, by organizing local and global events or supporting initiatives to make it more usable: interoperability, training, certification, trademark licenses...

Not all Foundations are the same

At this point it's important to see that a foundation is not a label, the name doesn't come with any guarantee. All those foundations are actually very different, and you need to read the fine print to understand their goals or assess exactly how open they are.

On the upstream side, few of them actually let their open source project be completely run by their individual contributors, with elected leadership (one contributor = one vote, and anyone may contribute). That form of governance is the only one that ensures that a project is really open to individual contributors, and the only one that prevents forks due to contributors and project owners not having aligned goals. If you restrict leadership positions to appointed seats by corporate backers, you've created a closed pay-to-play collaboration, not an open collaboration ground. On the downstream side, not all of them accept individual members or give representation to smaller companies, beyond their founding members. Those details matter.

When we set up the OpenStack Foundation, we worked hard to make sure we created a solid, independent, open and meritocratic upstream side. That, in turn, enabled a pretty successful downstream side, set up to be inclusive of the diversity in our ecosystem.

The future

I see the "Foundation" approach to open source as the only viable solution past a given size and momentum around a project. It's certainly preferable to "open but actually owned by one specific party" (which sooner or later leads to forking). Open source now being the default development model in the industry, we'll certainly see even more foundations in the future, not less.

As this approach gets more prevalent, I expect a rise in more tactical foundations that primarily exist as a trade association to push a specific vision for the industry. At OSCON during those two presentations around container-driven foundations, it was actually interesting to notice not the common points, but the differences. The message was subtly different (pods vs. containers), and the companies backing them were subtly different too. I expect differential analysis of Foundations to become a thing.

My hope is that as the "Foundation" model of open source gets ubiquitous, we make sure that we distinguish those which are primarily built to sustain the needs or the strategy of a dozen of large corporations, and those which are primarily built to enable open collaboration around an open source project. The downstream goal should stay a secondary goal, and new foundations need to make sure they first get the upstream side right.

In conclusion, we should certainly welcome more Foundations being created to sustain more successful open source projects in the future. But we also need to pause and read the fine print: assess how open they are, discover who ends up owning their upstream open source project, and determine their primary reason for existing.

New OpenStack component versioning

Yesterday we reached the liberty-1 development milestone. You may have noticed from the announcement that the various components released were all using new, different version numbers. What's going on here ?

Once upon a time

Since the beginning of OpenStack we've been using two versioning schemes. One was for projects released once every 6 months and following a schedule of development milestones and release candidates. Those would be using a YEAR.N version number (like 2015.1 for Kilo).

Another was used by Swift, which was already mature when OpenStack started, and which released intermediary versions as-needed throughout the cycle. It would use a X.Y.Z version number which looked a lot more like semantic versioning.

At the end of the cycle, we would coordinate a final release that would combine both. For example the "Kilo" release would be made of Nova 2015.1.0, Swift 2.3.0, and everything else at 2015.1.0.

Recent developments

A few things happened over the last two cycles. First, we released more and more libraries, and those would follow a strict X.Y.Z semantic versioning. Those would also have an final release in the cycle, from which a stable branch would be maintained for critical bugfixes and vulnerability fixes. So the portion of commonly-versioned YEAR.N deliverables was fast decreasing.

Second, some projects got more mature and/or more able to release fully-functional intermediary releases as-needed. As a community, we still can't support more than one stable branch every 6 months, so those intermediary releases won't get backports, but past a given maturity step, it's still a great thing to push new features to bleeding-edge users as early and often as we can. For those a YEAR.N synchronized versioning scheme would not work.

The versioning conundrum

At that stage we had three options to handle those projects switching from one model to another. They could keep their 2015.2.0 version and start doing semantic versioning from that -- but that would be highly confusing, when you end up releasing 2017.9.4 sometimes in 2016. The second option was to reset the version for projects as they switch. So Ironic would adopt, say, 3.0.0 while all other projects still use 2015.2.0.

The third option was to bite the bullet and drop the YEAR.N versioning at the same time, for all the projects that were still using it. Switching them all to some arbitrary number (say, 12.0.0 since that would be the 12th OpenStack release) would create confusion as projects switching to intermediary releases would slowly drift from the pack (most projects publishing 13.0.0 while some would be at 12.5.2 and others at 13.1.0). So to avoid that confusion, projects would pick purposefully distinct version numbers based on their age.

The change

After discussions at the Vancouver Design Summit and on the mailing-list, we opted for the third option, with an initial number calculated from the number of past integrated releases already published.

It's a clean cut which will reduce on-going disruption. All components end up with a different, meaningful version number: there are no longer "normal" and "outliers" projects. Additionally, it solves the weird impression we had when we released 2014.2.2 stable versions sometimes in 2015.

As far as impact is concerned, distributions will need to make sure to insert an epoch so that package versions sort correctly in their package management systems. If your internal CI pipeline relies on sorting version numbers, it will likely need an adjustment too. For everyone else, it should not have an impact: when Liberty is out, you will upgrade to the liberty version of the components, as you always did.

Liberty-1 and the future

The change in versions was pushed last week, and that is why for liberty-1 we published for Nova, for Keystone, and for Designate, etc... Those are still on a milestone-based 6-month release cycle, but their "Liberty" final version won't be all versioned "2015.2.0", but 12.0.0 for Nova, 8.0.0 for Keystone, etc.

To reduce the confusion, the release management team will provide tooling and web pages to describe what each series means in terms of component version numbers (and the other way around).

We hope this future-proof change will bring some more freedom for OpenStack project teams to pick the release model that is the most interesting for them and their user base. For a cycle named "liberty", that sounded like a pretty good time to do it.

OpenStack Technical Committee candidates

The election process to renew half of the OpenStack Technical Committee will start tomorrow with candidates self-nominating to run for election. As the chair of the existing Technical Committee (and running for reelection) I would like to share some thoughts on what would make in my opinion good TC members for the upcoming cycle.

A few words on the OpenStack Technical Committee role first. The role of the TC is to lead the "OpenStack" software development in general. Each individual project is lead by its project team and its PTL, but the TC leads "OpenStack" development. That includes defining the limits of what is considered "an OpenStack project": during the Kilo cycle we introduced new rules to handle that question, mostly based on alignment with the OpenStack Mission and determining if the new project team shares the common values OpenStack has been built on, behaves like an OpenStack project and therefore should be considered part of the OpenStack community. The role of the TC also includes providing guidance and advice to OpenStack projects, as well as driving horizontal efforts and solving cross-project issues.

The Technical Committee also serves as an ultimate appeals board to solve conflicts in our open source community. Its members are elected by all the OpenStack contributors and are trusted to make the right call should an issue escalate to them.

With all this said, what makes a good Technical Committee member ? I'd say the first attribute of a good candidate is how skilled they are and how respected their opinion is on the above topics. Do you trust them to make the right call as to what should be considered part of the OpenStack community ? Do you trust them to place "OpenStack" interest above specific projects (or specific companies) interest ? Do you trust their open source community experience to make the right judgment call in case issues are escalated ?

The other attribute is, I think, equally important. It's how much time you can specifically dedicate to the task of being a TC member. Being a good TC member takes a lot of time. It's not an honorific position where you happen to sit in a meeting one hour per week and decide the fate of OpenStack. It's about diving into proposals, look into new projects, drive cross-project initiatives, identify long-standing issues in our community and raise them. I spend about 40% of my work time working on TC stuff, and I wish I could spend more.

Let's be clear: I don't think that all TC members in the past had enough time to dedicate to Technical Committee work. They were all community leaders, skilled and trustworthy. But a lot of them had other fulltime responsibilities too, like being the PTL of a large OpenStack project, or being a manager in a major company or startup. In the end that meant the past TC members were not as active on cross-project matters as they could have been.

In liberty I'd like the Technical Committee to identify and propose plans to address long-standing issues in OpenStack in general. I'd like individual members to dive into specific projects and provide an audit of their current health. I'd like us to make things better, rather than limit ourselves to codifying governance. I'd like us to step out of the way, and start being more useful. That takes a lot of time.

So as self-nomination period opens, I would like to encourage new people to run for election. I'd like candidates to explain why they are fit for the job, and why they think they will have a lot of time for it. And I'd like voters to take that availability into account when they decide who to vote for.

Stepping out of the way

In the early days of OpenStack, we instituted a do-acracy: power to people that do things over people that don't. Code talks: when unsure on the direction to go, the one that came with code basically won. Don't ask for permission, ask for forgiveness. Make progress, fix issues if they arise. That let us positively move forward fast in the early days.

After some time, our community realized this didn't necessarily scale, and this didn't necessarily result in quality. As we kept on adding contributors way beyond our initial trusted small circle, we couldn't rely solely on our common culture and shared understandings anymore. We started introducing safeguards, the most visible and successful one being our code gating system (which relies on human code reviews and automated tests to prevent unacceptable code from merging). We started writing down our rules and governance, since our ever-increasing group couldn't rely on oral tradition and shared experience anymore. As picky newcomers complained about governance rules holes in every gray area, we started defining the process to define process.

I think we struck the right balance on those days between allowing our community to grow and letting people do things. But the movement to add rules unfortunately didn't stop then, it continues today. We added specs, which if used with moderation (like only for significant features) are a great way to avoid wasted development effort, but used blindly are a great way to increase the length of our feature development pipeline and increase frustration. We added automated tests to verify the presence (or is it absence ?) of a trailing dot in commit message titles. We started to get in the way of getting things done.

At the Technical Committee level, that translates into hours lost rubberstamping stuff, hours we don't spend asking ourselves the right questions (like: what is broken in OpenStack today and how to fix it). Do we really need to approve a project team decision to add a new git repository ? Or should we just let them do things, and consider reverting that action if it ends up causing a problem ? It is an interesting balance to strike between letting people go wild and approving their every move. On one side you don't want them to waste energy on something you might end up striking down, but on another side you don't want to block them when they first set out to create stuff.

As of today, I think we pushed the regulation and "ask for permission" cursor so far we actually prevent things from happening. I'd like us to step out of the way. When a proposal is not perfect, I'd like us to propose a subsequent patchset instead of blocking the review forever. At the Technical Committee level, I'd like us to let people do more things, and retreat to being an appeals board in case problems end up arising. The Technical Committee always has been the ultimate appeals board in case problems can't find a resolution at lower levels of our community. In practice, the mere existence of that appeals board encouraged conflict resolution at lower levels, to the point where I can't remember us being called to resolve an actual dispute. I'd like our community to be more trusted by default, to ask more for forgiveness and less for permission.

Trust is a weird thing. When your behavior is constrained by rules and automated tooling, you tend to try to game the system and get the most you can of it. But when trust is placed on you to do the right thing, the incentive is to avoid abusing the trust that is placed on you, to prove yourself worthy of that trust.

So I'd like us (our community in general and the Technical Committee in particular) to look into our processes and see where we can remove ourselves from the action pipeline. Where we can trust by default and rely on our escalation mechanisms to resolve issues as they arise (if they arise). That may come out as weird coming from a process/governance wonk like me, but that is my new motto for the Liberty cycle: "step out of the way".