We are 5 years into the OpenStack ride (and 3 years into the OpenStack
Foundation ride), and the challenges for our community are evolving.
In this article I want to talk about what I consider the most significant
threat for our open source community today: the loss of our common culture.
Over the past year we evolved the OpenStack project model to adopt an
inclusive approach. Project teams which
work on deliverables that help us achieve the OpenStack Mission, and which
follow our development and community practices should generally be accepted
under the "big tent". As we explained in
this presentation in Vancouver,
we moved from asking "is this OpenStack ?" to asking "are you OpenStack ?".
What does it mean to be OpenStack ? We wrote down a
set of principles,
based on the original four opens
that were defined at the very beginning of this journey. But
"being OpenStack" goes beyond that. It is to be aligned on a common goal,
be part of the same effort, be the same tribe. OpenStack relies on a number
of individuals working cross-project (on infrastructure, QA, documentation,
release processes, interoperability, user experience, API guidelines,
vulnerability management, election organization...). It is because we belong
to the same tribe that some people and organizations care enough about
"OpenStack" as a whole to dedicate time to those essential cross-project
efforts.
This is why we standardize on logged IRC channels as a communication medium,
why we ask that every project change goes through Gerrit, and why we should
very conservatively add new programming languages to the mix. Some people
advocate letting OpenStack project teams pick whatever language they want,
or letting them meet on that new trendy videoconferencing app, or letting
them track bugs on separate JIRA instances. More freedom sounds good at first
glance, but doing so would further fragment our community into specific silos
that all behave differently. Doing so would make recruiting for those
essential cross-project efforts even harder than it is today, while at the
same time making the work of those cross-project efforts significantly more
complex. Doing so would make our community crumble under its own weight.
We started this journey with a pretty strong common culture. It was mostly
oral tradition. We assumed that as OpenStack grew, our culture would
naturally be assimilated by new members. And it did, for quite some time.
But today we are at a point where we dramatically expanded our community
(we doubled the number of project teams over the last year) and our common
culture did not naturally transmit to newcomers. Silos with local traditions
have formed. Teams don't all behave in the same way anymore. Most team
members only care about a single project team. We struggle to move from one
project to another. We struggle to provide common solutions that work for
everyone. We struggle to recruit for cross-project efforts more than we ever
did. OpenStack's future as a community is at risk. It's time to hold the
culture line, rather than time to further relax it.
It is also more than time that we document our common culture, so that it
can be explicitly communicated to everyone in the OpenStack ecosystem
(current and prospective members). We started a workgroup at the
Technical Committee, held a
virtual sprint to get a
base version written, and now here it is: the first version of the
OpenStack Project team guide.
Read it, refer to it, communicate it to your OpenStack community fellows,
propose changes to it. It is an essential tool for us to overcome this new
challenge. It's certainly not the only tool, and I hope we'll be able to
dedicate a cross-project session at the Mitaka Design Summit in Tokyo to
further discuss this topic.
At OSCON last week, Google announced the creation around Kubernetes of the
Cloud-Native Computing Foundation. The next day, Jim Zemlin dedicated his
keynote to the (recently-renamed) Open Container Initiative, confirming the
Linux Foundation's recent shift towards providing Foundations-as-a-Service.
Foundations ended up being the talk of the show, with some questioning the
need for Foundations for everything, and others discussing the rise of
Foundations as tactical weapons.
Back to the basics
The main goal of open source foundations is to provide a neutral, level and
open collaboration ground around one or several open source projects. That is
what we call the upstream support goal. Projects are initially created by
individuals or companies that own the original trademark and have power to
change the governance model. That creates a tilted playing field: not all
players are equal, and some of them can even change the rules in the middle
of the game. As projects become more popular, that initial parentage becomes
a blocker for other contributors or companies to participate. If your goal is
to maximize adoption, contribution and mindshare, transferring the ownership
of the project and its governance to a more neutral body is the natural next
step. It removes barriers to contribution and truly enables open innovation.
Now, those foundations need basic funding, and a common way to achieve that
is to accept corporate members. That leads to the secondary goal of open source
foundations: serve as a marketing and business development engine for companies
around a common goal. That is what we call the downstream support goal.
Foundations work to build and promote a sane ecosystem around the open source
project, by organizing local and global events or supporting initiatives to
make it more usable: interoperability, training, certification, trademark
licenses...
Not all Foundations are the same
At this point it's important to see that a foundation is not a label, the
name doesn't come with any guarantee. All those foundations are actually very
different, and you need to read the fine print to understand their goals or
assess exactly how open they are.
On the upstream side, few of them actually let their open source project be
completely run by their individual contributors, with elected leadership
(one contributor = one vote, and anyone may contribute). That form of
governance is the only one that ensures that a project is really open to
individual contributors, and the only one that prevents forks due to
contributors and project owners not having aligned goals. If you restrict
leadership positions to appointed seats by corporate backers, you've created
a closed pay-to-play collaboration, not an open collaboration ground. On the
downstream side, not all of them accept individual members or give
representation to smaller companies, beyond their founding members.
Those details matter.
When we set up the OpenStack Foundation, we worked hard to make sure we
created a solid, independent, open and meritocratic upstream side. That,
in turn, enabled a pretty successful downstream side, set up to be inclusive
of the diversity in our ecosystem.
The future
I see the "Foundation" approach to open source as the only viable solution
past a given size and momentum around a project. It's certainly preferable
to "open but actually owned by one specific party" (which sooner or later
leads to forking). Open source now being the default development model in
the industry, we'll certainly see even more foundations in the future, not
less.
As this approach gets more prevalent, I expect a rise in more tactical
foundations that primarily exist as a trade association to push a specific
vision for the industry. At OSCON during those two presentations around
container-driven foundations, it was actually interesting to notice not the
common points, but the differences. The message was subtly different (pods
vs. containers), and the companies backing them were subtly different too.
I expect differential analysis of Foundations to become a thing.
My hope is that as the "Foundation" model of open source gets ubiquitous,
we make sure that we distinguish those which are primarily built to sustain
the needs or the strategy of a dozen of large corporations, and those which
are primarily built to enable open collaboration around an open source project.
The downstream goal should stay a secondary goal, and new foundations need
to make sure they first get the upstream side right.
In conclusion, we should certainly welcome more Foundations being created to
sustain more successful open source projects in the future. But we also need
to pause and read the fine print: assess how open they are, discover who ends
up owning their upstream open source project, and determine their primary
reason for existing.
Yesterday we reached the liberty-1 development milestone. You may have noticed
from the announcement
that the various components released were all using new, different version
numbers. What's going on here ?
Once upon a time
Since the beginning of OpenStack we've been using two versioning schemes.
One was for projects released once every 6 months and following a schedule
of development milestones and release candidates. Those would be using
a YEAR.N version number (like 2015.1 for Kilo).
Another was used by Swift, which was already mature when OpenStack started,
and which released intermediary versions as-needed throughout the cycle. It
would use a X.Y.Z version number which looked a lot more like semantic
versioning.
At the end of the cycle, we would coordinate a final release that would
combine both. For example the "Kilo" release would be made of Nova 2015.1.0,
Swift 2.3.0, and everything else at 2015.1.0.
Recent developments
A few things happened over the last two cycles. First, we released more and
more libraries, and those would follow a strict X.Y.Z semantic versioning.
Those would also have an final release in the cycle, from which a stable
branch would be maintained for critical bugfixes and vulnerability fixes. So
the portion of commonly-versioned YEAR.N deliverables was fast decreasing.
Second, some projects got more mature and/or more able to release
fully-functional intermediary releases as-needed. As a community, we still
can't support more than one stable branch every 6 months, so those intermediary
releases won't get backports, but past a given maturity step, it's still a
great thing to push new features to bleeding-edge users as early and often as
we can. For those a YEAR.N synchronized versioning scheme would not work.
The versioning conundrum
At that stage we had three options to handle those projects switching from
one model to another. They could keep their 2015.2.0 version and start doing
semantic versioning from that -- but that would be highly confusing, when you
end up releasing 2017.9.4 sometimes in 2016. The second option was to reset
the version for projects as they switch. So Ironic would adopt, say, 3.0.0
while all other projects still use 2015.2.0.
The third option was to bite the bullet and drop the YEAR.N versioning at the
same time, for all the projects that were still using it. Switching
them all to some arbitrary number (say, 12.0.0 since that would be the 12th
OpenStack release) would create confusion as projects switching to
intermediary releases would slowly drift from the pack (most projects
publishing 13.0.0 while some would be at 12.5.2 and others at 13.1.0).
So to avoid that confusion, projects would pick purposefully distinct version
numbers based on their age.
The change
After discussions at the Vancouver Design Summit and on the mailing-list,
we opted for the third option, with an initial number calculated from the
number of past integrated releases already published.
It's a clean cut which will reduce on-going disruption. All components end
up with a different, meaningful version number: there are no longer "normal"
and "outliers" projects. Additionally, it solves the weird impression we had
when we released 2014.2.2 stable versions sometimes in 2015.
As far as impact is concerned, distributions will need to make sure to insert
an epoch so that package versions sort correctly in their package management
systems. If your internal CI pipeline relies on sorting version numbers, it
will likely need an adjustment too. For everyone else, it should not have an
impact: when Liberty is out, you will upgrade to the liberty version of the
components, as you always did.
Liberty-1 and the future
The change in versions was pushed last week, and that is why for liberty-1 we
published 12.0.0.0b1 for Nova, 8.0.0.0b1 for Keystone, and 1.0.0.0b1 for
Designate, etc... Those are still on a milestone-based 6-month release cycle,
but their "Liberty" final version won't be all versioned "2015.2.0", but 12.0.0
for Nova, 8.0.0 for Keystone, etc.
To reduce the confusion, the release management team will provide tooling and
web pages to describe what each series means in terms of component version
numbers (and the other way around).
We hope this future-proof change will bring some more freedom for OpenStack
project teams to pick the release model that is the most interesting for them
and their user base. For a cycle named "liberty", that sounded like a pretty
good time to do it.
The election process to renew half of the OpenStack Technical Committee will
start tomorrow with candidates self-nominating to run for election. As the
chair of the existing Technical Committee (and running for reelection) I would
like to share some thoughts on what would make in my opinion good TC members
for the upcoming cycle.
A few words on the OpenStack Technical Committee role first. The role of the
TC is to lead the "OpenStack" software development in general. Each individual
project is lead by its project team and its PTL, but the TC leads "OpenStack"
development. That includes defining the limits of what is considered "an
OpenStack project": during the Kilo cycle we introduced new rules to handle
that question, mostly based on alignment with the OpenStack Mission and
determining if the new project team shares the common values OpenStack has been
built on, behaves like an OpenStack project and therefore should be considered
part of the OpenStack community. The role of the TC also includes providing
guidance and advice to OpenStack projects, as well as driving horizontal
efforts and solving cross-project issues.
The Technical Committee also serves as an ultimate appeals board to solve
conflicts in our open source community. Its members are elected by all the
OpenStack contributors and are trusted to make the right call should an issue
escalate to them.
With all this said, what makes a good Technical Committee member ? I'd say
the first attribute of a good candidate is how skilled they are and how
respected their opinion is on the above topics. Do you trust them to make the
right call as to what should be considered part of the OpenStack community ?
Do you trust them to place "OpenStack" interest above specific projects (or
specific companies) interest ? Do you trust their open source community
experience to make the right judgment call in case issues are escalated ?
The other attribute is, I think, equally important. It's how much time you can
specifically dedicate to the task of being a TC member. Being a good TC member
takes a lot of time. It's not an honorific position where you happen to sit
in a meeting one hour per week and decide the fate of OpenStack. It's about
diving into proposals, look into new projects, drive cross-project initiatives,
identify long-standing issues in our community and raise them. I spend about
40% of my work time working on TC stuff, and I wish I could spend more.
Let's be clear: I don't think that all TC members in the past had enough time
to dedicate to Technical Committee work. They were all community leaders,
skilled and trustworthy. But a lot of them had other fulltime responsibilities
too, like being the PTL of a large OpenStack project, or being a manager in
a major company or startup. In the end that meant the past TC members were
not as active on cross-project matters as they could have been.
In liberty I'd like the Technical Committee to identify and propose plans
to address long-standing issues in OpenStack in general. I'd like individual
members to dive into specific projects and provide an audit of their current
health. I'd like us to make things better, rather than limit ourselves to
codifying governance. I'd like us to
step out of the way,
and start being more useful. That takes a lot of time.
So as self-nomination period opens, I would like to encourage new people to
run for election. I'd like candidates to explain why they are fit for the job,
and why they think they will have a lot of time for it. And I'd like voters to
take that availability into account when they decide who to vote for.
In the early days of OpenStack, we instituted a do-acracy: power to people
that do things over people that don't. Code talks: when unsure on the direction
to go, the one that came with code basically won. Don't ask for permission,
ask for forgiveness. Make progress, fix issues if they arise. That let us
positively move forward fast in the early days.
After some time, our community realized this didn't necessarily scale, and
this didn't necessarily result in quality. As we kept on adding contributors
way beyond our initial trusted small circle, we couldn't rely solely on our
common culture and shared understandings anymore. We started introducing
safeguards, the most visible and successful one being our code gating system
(which relies on human code reviews and automated tests to prevent unacceptable
code from merging). We started writing down our rules and governance, since
our ever-increasing group couldn't rely on oral tradition and shared experience
anymore. As picky newcomers complained about governance rules holes in every
gray area, we started defining the process to define process.
I think we struck the right balance on those days between allowing our
community to grow and letting people do things. But the movement to add rules
unfortunately didn't stop then, it continues today. We added specs, which if
used with moderation (like only for significant features) are a great way to
avoid wasted development effort, but used blindly are a great way to increase
the length of our feature development pipeline and increase frustration. We
added automated tests to verify the presence (or is it absence ?) of a
trailing dot in commit message titles. We started to get in the way of getting
things done.
At the Technical Committee level, that translates into hours lost
rubberstamping stuff, hours we don't spend asking ourselves the right
questions (like: what is broken in OpenStack today and how to fix it). Do we
really need to approve a project team decision to add a new git repository ?
Or should we just let them do things, and consider reverting that action if it
ends up causing a problem ? It is an interesting balance to strike between
letting people go wild and approving their every move. On one side you don't
want them to waste energy on something you might end up striking down, but on
another side you don't want to block them when they first set out to create
stuff.
As of today, I think we pushed the regulation and "ask for permission" cursor
so far we actually prevent things from happening. I'd like us to step out of
the way. When a proposal is not perfect, I'd like us to propose a subsequent
patchset instead of blocking the review forever. At the Technical Committee
level, I'd like us to let people do more things, and retreat to being an
appeals board in case problems end up arising. The Technical Committee always
has been the ultimate appeals board in case problems can't find a resolution
at lower levels of our community. In practice, the mere existence of that
appeals board encouraged conflict resolution at lower levels, to the point
where I can't remember us being called to resolve an actual dispute. I'd like
our community to be more trusted by default, to ask more for forgiveness and
less for permission.
Trust is a weird thing. When your behavior is constrained by rules and
automated tooling, you tend to try to game the system and get the most you
can of it. But when trust is placed on you to do the right thing, the
incentive is to avoid abusing the trust that is placed on you, to prove
yourself worthy of that trust.
So I'd like us (our community in general and the Technical Committee in
particular) to look into our processes and see where we can remove ourselves
from the action pipeline. Where we can trust by default and rely on our
escalation mechanisms to resolve issues as they arise (if they arise). That
may come out as weird coming from a process/governance wonk like me, but that
is my new motto for the Liberty cycle: "step out of the way".
