In a recent Technical Committee update on the OpenStack blog,
I explained how the OpenStack "integrated release" concept, due to its binary
nature, ended up meaning all sorts of different things to different people.
That is the main reason why we want to deconstruct its various meanings into
a set of tags that can be independently applied to projects, in order to more
accurately describe our project ecosystem.
In this blogpost, I'll look into the various meanings the "integrated release"
ended up having in our history, and see how we can better convey that
information through separate tags.
Released together
The original meaning of "integrated release" is that all the projects in it
are released together on the same date, at the end of our development cycles.
I recently proposed a tag
("release:at-6mo-cycle-end") to describe projects that commit to producing
a release at the end of our development cycles, which I think will cover
that facet.
Managed release
Very quickly, the "integrated release" also described projects which had their
release managed by the OpenStack Release Management team. This team sets up
processes (mostly a
Feature Freeze
deadline and a
release candidate dance)
to maximize the chances that managed projects would release on a pre-announced
date, and that none of the managed projects would end up delaying the end
release date.
Projects would be added to incubation and when we thought they were ready to
follow those processes without jeopardizing the "integrated release" for
existing projects, they would get added to be the next release.
That is a separate facet from the previous one, so I
proposed a separate tag
("release:managed") to describe projects that happen to still be handled by
the Release Management team.
Co-gating
As we introduced complex code gating in OpenStack Infrastructure, the
"integrated release" concept grew another facet: it would also mean that
projects changes are tested against one another master branches. That way
we ensure that changes in project B don't break project A. This is rather
neat for a tightly-coupled set of projects. But this may be a bit overkill
for projects higher in the stack that just consume public APIs. Especially
when non-deterministic test errors in project A prevent changes from landing
in B.
We need to revise how to split co-gating in meaningful groups.
This work is led by Sean Dague.
Once it is completed, I expect us to convey the information on what is
actually tested together using tags as well.
Supported by OpenStack horizontal efforts
From there, the "integrated release" naturally evolved to also being the set
of projects that horizontal efforts (such as Documentation, QA, stable branch
maintenance, Vulnerability management, Translations...) would focus their
work on. Being part of the "integrated release" ensured that you would get
fully supported by those horizontal teams.
That didn't scale that well, though. The documentation team was the first to
exhaust their limited resources, and started to move to a model where they
would not directly write all the docs for all the integrated projects. Since
then, all horizontal teams decided to gradually move to the same model, where
they would directly handle a number of projects (or none), but provide tooling,
mentoring and support for all the others.
It's still valuable information to know which project happens to be directly
handled by which horizontal effort, and which project ends up having security
support or basic docs. So we'll introduce tags in the future to accurately
describe this facet.
The base features
Going downhill, the "integrated release" started to also mean the base
features you can rely on being present when people say they deployed
"OpenStack". That was a bit at odds with the previous facets though: why would
all co-gating projects with a coordinated release necessarily be essential ?
And indeed, the integrated release grew beyond "base" features (like Keystone)
to include obviously optional projects (like Sahara or Ceilometer).
I personally think our users would still benefit from a description of what
layer each project belongs to: is it providing base IaaS compute functionality,
or is it relying on that being present ? This is not a consensual view though,
as some people object to all proposals leading to any type of ranking within
OpenStack projects.
OpenStack
At that point, the "integrated release" was "the OpenStack release", and things
outside of it were "not OpenStack".
This obviously led to more pressure to add more projects in it. But when the
OpenStack governance (previously under a single Project Policy Board
banner) was split between the Technical Committee and the Foundation Board,
the former retained control over the "integrated release" contents, and the
latter took control of trademark usage. This created tension over that specific
facet.
Defcore was created to solve this problem, by defining the criteria to apply
to various trademark programs. When asked to provide a set of projects (or
rather, as set of sections of code) to apply the trademark on, the Technical
Committee answered with the only concept it had (you guessed it, "the
integrated release" again).
In the tags world, when asked for a set of projects to apply a particular
trademark program to, the Technical Committee shall be able to provide a
finer-grained answer, by defining a specific tag for each question.
Stability
Further downhill, "integrated release" also started to mean "stable". Once
published in such a release, a project would not remove a feature without
proper discussion, notice, and deprecation period. That was yet another
facet of the now-bloated "integrated release" concept.
The issue is, all projects were not able to commit to the same stability
rules. One would never deprecate an existing feature, while one would rip
its API off over the course of two development cycles. One size didn't fit
all.
In the tags world, my view is that Ops and devs, working together, should
define various stability levels and the rules that apply for each. Then each
project can pick the tag corresponding to the stability level they can commit
to.
Maturity
Last but not least, at one point people started to assume that projects in
the "integrated release" were obviously mature. They were all in widespread
usage, enterprise-ready, carrier-grade, service-provider-class, web-scale.
The reality is, this facet is also complex: some projects are, and some
projects are less, and some projects aren't. So we need to
describe the various maturity styles and levels, and inform our users of each
project real status.
It's difficult to describe maturity objectively though. I intend to discuss
that topic with the ones that are the best placed to accurately describe it:
the OpenStack operators gathered at the Ops Summit next week.
The OpenStack project structure has been under heavy discussion over the
past months. There was a
long email thread,
a
lot
of
opinionated
blogposts,
a cross-project design summit session
in Paris, and various strawmen proposed to our
governance repository.
Based on all that input, the OpenStack Technical Committee worked on a clear
specification of the problems we are trying to solve, and the proposed way
to fix them. What follows is an excerpt of the
approved resolution.
Problem description
Our project structure is currently organized as a ladder. Developers form
teams, work on a project, then apply for incubation and ultimately graduate
to be part of the OpenStack integrated release. Only integrated projects
(and the few horizontal efforts necessary to build them) are recognized
officially as "OpenStack" efforts. This creates a number of issues, which
were particularly visible at the Technical Committee level over the Juno
development cycle.
First, the integrated release as it stands today is not a useful product for
our users. The current collection of services in the integrated release spans
from cloud native APIs (swift, zaqar in incubation), base-level IaaS blocks
(nova, glance, cinder), high-level aaS (savana, trove), and lots of things
that span domains. Some projects (swift, ironic...) can be used quite well
outside of the rest of the OpenStack stack, while others (glance, nova)
really don't function in a different context. Skilled operators aren't
deploying "the integrated release": they are picking and choosing between
components they feel are useful. New users, however, are presented with a
complex and scary "integrated release" as the thing they have to deploy and
manage: this inhibits adoption, and this inhibits the adoption of a slice of
OpenStack that could serve their need.
Second, the integrated release being the only and ultimate goal for projects,
there is no lack of candidates, and the list is always-growing. Why reject
Sahara if you accepted Trove? However, processes and services are applied
equally to all members of the integrated release: we gate everything in the
integrated release against everything else, we do a common, time-based
release every 6 months, we produce documentation for all the integrated
release components, etc. The resources working on those integrated horizontal
tasks are very finite, and complexity grows non-linearly as we add more
projects. So there is outside pressure to add more projects, and internal
pressure to resist further additions.
Third, the binary nature of the integrated release results in projects
outside the integrated release failing to get the recognition they deserve.
"Non-official" projects are second- or third-class citizens which can't get
development resources. Alternative solutions can't emerge in the shadow of
the blessed approach. Becoming part of the integrated release, which was
originally designed to be a technical decision, quickly became a
life-or-death question for new projects, and a political/community minefield.
In summary, the "integrated release" is paradoxically too large to be
effectively integrated, installed or upgraded in one piece, and too small to
express the diversity of our rich ecosystem. Its slow-moving, binary nature
is too granular to represent the complexity of what our community produces,
and therefore we need to reform it.
The challenge is to find a solution which allows to address all those three
issues. Embrace the diversity of our ecosystem while making sure that what
we produce is easily understandable and consumable by our downstream users
(distributions, deployers, end users), all that without putting more stress
on the already overworked horizontal teams providing services to all
OpenStack projects, and without limiting the current teams access to common
finite resources.
Proposed change
Provide a precise taxonomy to help navigating the ecosystem
We can't add any more "OpenStack" projects without dramatically revisiting
the information we provide. It is the duty of the Technical Committee to
help downstream consumers of OpenStack understand what each project means
to them, and provide them with accurate statuses for those projects.
Currently the landscape is very simple: you're in the integrated release, or
you're not. But since there was only one category (or badge of honor), it
ended up meaning different things to different people. From a release
management perspective, it meant what we released on the same day. From a
CI perspective, it meant what was co-gated. From an OpenStack distribution
perspective, it meant what you should be packaging. From some operator
perspective, it meant the base set of projects you should be deploying. From
some other operator perspective, it meant the set of mature, stable projects.
Those are all different things, and yet we used a single category to describe
it.
The first part of the change is to create a framework of tags to describe
more accurately and more objectively what each project produced in the
OpenStack community means. The Technical Committee will define tags and the
objective rules to apply them. This framework will allow us to progressively
replace the "integrated release" single badge with a richer and more nuanced
description of all "OpenStack" projects. It will allow the Technical
Committee to provide more precise answers to the Foundation Board of
Directors questions about which set of projects may make sense for a given
trademark license program. It will allow our downstream users to know which
projects are mature, which are security-supported, which are used in more
than one public cloud, or which are really massively scalable.
Recognize all our community is a part of OpenStack
The second part of the change is recognizing that there is more to
"OpenStack" than a finite set of projects blessed by the Technical
Committee. We already have plenty of projects that are developed on
OpenStack infrastructure, follow the OpenStack way of doing things, have
development discussions on the openstack-dev mailing-list and
use #openstack-meeting channels for their team meetings. Those are part of
the OpenStack community as well, and we propose that those should considered
"OpenStack projects" (and be hosted under openstack git namespaces), as
long as they meet an objective criteria for inclusion (to be developed as one
of the work items below). This might include items such as:
-
They align with the OpenStack Mission: the project should help further the
OpenStack mission, by providing a cloud infrastructure service, or
directly building on an existing OpenStack infrastructure service
-
They follow the OpenStack way: open source (licensing), open community
(leadership chosen by the contributors to the project), open development
(public reviews on Gerrit, core reviewers, gate, assigned liaisons), and
open design (direction discussed at Design Summit and/or on public forums)
-
They ensure basic interoperability (API services should support at least
Keystone)
-
They submit to the OpenStack Technical Committee oversight
These criteria are objective, and therefore the Technical Committee may
delegate processing applications to another team. However, the TC would
still vote to approve or reject applications itself, based on the
recommendations and input of any delegates, but without being bound to
that advice. The TC may also decide to encourage collaboration between
similar projects (to reduce unnecessary duplication of effort), or to
remove dead projects.
This proposed structure will replace the current program-driven structure.
We'll still track which team owns which git repository, but this will let
multiple different "OpenStack" teams potentially address the same problem
space. Contributors to projects in the OpenStack git namespace will all be
considered ATCs and participate in electing the Technical Committee.
Transition
As for all significant governance changes, we need to ensure a seamless
transition and reduce the effect of the reform on the current development
cycle. To ensure this seamless transition, the OpenStack taxonomy will
initially define one tag, "integrated-release", which will contain the
integrated projects for the Kilo cycle. To minimize disruption, this tag
will be used throughout the Kilo development cycle and for the Kilo end
release. This tag may be split, replaced or redefined in the future, but
that will be discussed as separate changes.
Next steps
I invite you to read
the full text
of this Technical Committee resolution to learn more about the proposed
implementation steps or the impact on current projects.
It's important to note that most of the work and decisions are still ahead of
us: those proposed changes are just the base foundational step, enabling the
future evolution of our project structure and governance. Nevertheless, it
still is a significant milestone to clearly describe the issues we are
working to solve, and to agree on a clear way forward to fix those.
The next step now is to communicate more widely about the direction we are
going, and start the discussion on some more difficult and less consensual
details (like the exact set of objective rules applied to judge new entrants,
or the need - and clear definition for - a compute-base tag).
I'm extremely excited by the coming OpenStack Summit week in Paris !
First, well, Paris. It is a pretty amazing city, with great sights, museums and
restaurants. Those who arrived early should have an awesome Saturday, sun all
day, 18°C/65°F high. This is pretty rare in November, so we are very lucky !
Then next week gets cold and wet, complete with a traditional transport
strike on Tuesday, so we don't mind being stuck all day in windowless design
summit rooms... Perfect!
Second, this OpenStack Summit is completely sold out days before the event.
That means we managed to get to a critical mass of local/regional community.
That wasn't a given: when we started working on a summit in Paris two years
ago, we were betting on the promising signs that Europe in general, and France
in particular, showed at that time. It feels like we are at the right place
at the right moment after all.
Third, our sponsors and party organizers seem to have outdone themselves. The
bar was set pretty high by previous events. But even if I lived in this city
for a long time, I'm very interested in participating to the pretty exclusive
events that were organized as social events for this week... In particular,
the Tuesday party at Pavillons de Bercy is, in my honest opinion, not something
anyone should miss.
Fourth and finally, 4 years and 10 releases in, OpenStack is at a crossroads.
There is plenty of past to reflect on, and plenty of future to discuss. There
are a number of processes and structures we need to evolve. There are a number
of changes we need to adapt to. I expect this week to be critical in shaping
our future successes: it's therefore pretty important to be around and
participate to the open discussion.
Shameless plug: I'll be moderating a
panel of old-timers on Wednesday at 9am, to reflect
back on what we achieved, and think about what we failed to achieve in these
last four years. Even if you were at the party the night before, that should
give you a good reason to get up !
So please, all enjoy this great week coming up.
Unless you have been living under a rock, you must have noticed that the
OpenStack community is currently brainstorming potential evolutions
to its project and governance structure. A lot of those posts focused on
solutions and implementation details, and it's easy to design the wrong
solution when you didn't first define the problem space. So in this post I'd
like to take a step back and look at which issues are we trying to solve, and
the constraints we have.
With the continued growth of OpenStack, the current project structure and
governance model (which you can largely blame me for) fail in various ways.
The "integrated release" Grail
In our current structure, the integrated release is the ultimate goal, with
being recognized as an official program and being incubated seen as the first
steps in the enlightenment ladder. That creates a number of issues.
First, the integrated release being the ultimate goal, there is no lack of
candidate projects, and it is always-growing. Why reject Sahara if you
accepted Trove ? However, processes and services are applied equally to all
members of the integrated release: we gate everything in the integrated
release against everything else, we do a common, time-based release every
6 months, we produce documentation for all the integrated release components,
etc. The resources working on those integrated horizontal tasks are very
finite, and complexity grows non-linearly as we add more projects. So there
is outside pressure to add more projects, and internal pressure to resist
further additions. This is obviously not sustainable.
Second, projects outside the integrated release fail to get the recognition
they deserve. Some companies won't invest resources to participate in a
"non-official" project. So becoming part of the integrated release, which
was designed to be a technical decision, quickly became a life-or-death
question for new projects, and a political/community minefield.
We need to find a model that lets us be inclusive of all "OpenStack" projects,
while preserving the resources of the horizontal teams. I think Monty's
"layer #1" (which I prefer to call Ring0) solves that issue, by defining
a user-case-driven, limited, mostly-static set that the current horizontal
teams (infra, release management, QA, Docs...) can commit to supporting
directly. All the other projects may or may not be supported directly (or
just getting tools and advice to do it themselves), depending on those
horizontal teams capacity. By making ring0 arbitrarily small and static, most
things would not be in it, so we avoid the Grail effect. Projects outside
ring0 can enjoy more freedom (with their release cycle, with their gating
choices...) than the tightly-controlled ring0. Ring0 becomes a production
artifact, not a badge of honor.
Duplication, Competition and Overlap
In the early days of this project, I was obsessed with avoiding duplication
of effort. In a large project like OpenStack, it's really easy for teams to
work in their (usually corporate) corner on their own invented-here solution
for a common problem. So the governance model was built to reduce the risk of
duplication of effort, by creating obviously-blessed projects, obsessively
avoiding overlap in scope, and constantly encouraging people to merge their
work and join existing teams rather than do their own thing. We could say it
was quite successful, and whoever has attended our Design Summits can see
how this cross-organization collaboration miracle happens every time.
Some take it now for granted, but it didn't happen by accident.
That said, there is now the unintended side-effect that we actively prevent
the emergence of better alternatives to replace existing "blessed" things.
It's currently difficult for a project to prove itself outside an official
program, in the shadow of an existing project with the "integrated" badge.
It's a difficult balance to strike. We still mostly want to avoid duplication
of effort and have teams cooperate and join efforts on the same code base,
since avoiding such waste is one of the big benefits of our open source/open
design/open development model. But we still want to let new flowers bloom.
I hope that removing blessed programs and the ladder to integrated will go
a long way to fix this, but I'd hate it if we ended up discouraging
collaboration as a side effect.
We also need to keep discouraging partially-overlapping scopes, because that
would be a disservice to our users. If a project does AB and another does CD,
there is little value to our users in a new project that would do ABC.
Not answering the right questions
The last issue with the current system is that it fails to answer the
questions that our downstream users (packagers, deployers, end users) rightly
ask themselves about OpenStack. A single, binary badge of honor just can't
answer all the different questions. Jay's
post
touches on that part extensively: I think developing a tag taxonomy to
document what each project in "OpenStack" can do for you, depending on what
type of user you are, is a good answer. Removing the "integrated release"
super-badge will allow that information to emerge.
The limit of the tent
So it's quite obvious at this point that we need to change the concept of
"integrated release", which is a lot too binary. Replacing it with a set of
flexible structures (a tag taxonomy, a ring0 that current horizontal teams in
OpenStack feel fine supporting, other groups as needed...) sounds like the
way to go. Being inclusive in what we accept in the "big tent" is also very
consensual. But even big tents have limits, lines in the sand that we draw
between what is in and what is not in. Where should be ours ? Supporters of
the big tent approach all seem to diverge on that detail.
There is still a marginal cost to pay for each project we add. If we decide
to precisely describe projects in the OpenStack ecosystem using a set of tags,
as Jay suggested, then it's only valuable if it's kept current, and the
maintenance cost increases (linearly) with each project addition. If those
projects are to be called "OpenStack", they may trigger a trademark search,
too. And what about Design Summit space ?
I'd like projects to at very least loosely align with the OpenStack mission,
otherwise we'd lose our key identity and purpose. Monty proposes an alignment
check ("are you one of us"), which mostly translates into observing a number
of key governance and technical principles. He would also focus design summit
space on "Ring0" projects. I fear that may make Ring0 artificially attractive
and recreate a Grail effect, but I don't have a better solution to suggest.
The TC constituency
The last constraint would be to find the right constituency for the Technical
Committee. The Technical Committee is defined in the Foundation bylaws as the
governance body in charge of the technical direction of "OpenStack" as a whole.
There is an essential symmetry, where projects willingly place themselves
under the authority of the TC, and in exchange get the right to participate
in the Technical Committee members election. We used to do that by blessing
individual projects. Currently we bless teams instead (the "Programs" concept
was created to add a level of indirection between the team we bless and the
code repositories they want to work on, to give teams more flexibility on how
they organize their code). As we change the structure, we need to find what
the new area of authority of the TC is, and therefore what its new
constituency would look like.
There are two approaches. You can consider that the TC only has authority
over a subset (Ring0) and supporting projects (horizontal efforts which
apply to Ring0). But that kind of mean other projects are not really
"OpenStack". The other approach is to consider that all the "big tent" is
under the authority of the Technical Committee, and contributing to any
project in the big tent gives you a right to vote in TC membership elections.
I'm leaning toward the latter option, even if that will change the dynamics
of the TC and may require stronger alignment checks ("are you one of us")
before entry.
Conclusion
As we move on and prepare more detailed proposals, I'd like to make sure all
the proposed solutions have an answer for those various questions:
- How do we solve the integrated gate bottleneck ?
- How do we solve the horizontal teams scaling issues ?
- How do we prevent having a single Grail, to move the TC away from its
current badge-granting authority role ?
- How do we allow competition and alternative solutions ?
- How do we continue to encourage collaboration and avoid duplication of
effort ?
- How do we keep on discouraging partial scope overlap ?
- How do we provide the answers about our projects that our
packagers/deployers/endusers are needing ?
- If a "big tent" approach is proposed, where does the big tent end ?
- What would the Design Summit look like in the new world order ?
- What would the TC area of authority (and therefore constituency) be in
the new world order ?
Now, brainstorm on.
It's that time of the year again... As we get to the end of our development
cycle, some people look at hard contributions numbers and some companies
decide to expose questionable metrics to brag about being #1.
This cycle, it's apparently HP's turn.
But you can make numbers say anything... It's true that HP is the largest
contributor when you consider the integrated release, the incubated projects
and the supporting projects altogether, and we are very happy with HP's
involvement in OpenStack. But numbers can always tell a different story
if you look at them from a slightly different angle. If you only consider
commits to the integrated release (the projects that we will end up releasing
as "Juno" in a few weeks), then Red Hat is still #1. If you count code reviews
on this same integrated release, then Mirantis leads. If you only consider
Documentation, then SUSE leads. It's all pretty balanced and healthy.
Rather than celebrating being #1, we should all celebrate the diversity of our
community and our contributors. 132 different organizations were involved.
6% of our code contributions come from individuals who are not affiliated
with any company or organization. The
Universidade Federal de Campina Grande
is #25 in commits to OpenStack projects.
B1 systems, a small German consulting
firm, is #8, mostly through the work of one person.
Every little bit counts. It doesn't matter who is #1, it matters that we all
can contribute, and that we all do contribute.
It matters that we keep on making sure everyone can easily
contribute. That's what's really important, and I wish we all were celebrating
that.
