Since the beginning of March, OpenStack developers are focusing on
testing and bugfixes, with the objective of producing a release
candidate for each project.
Regular readers of my blog know I'm not the last one to complain when I
feel developers don't care about the release and don't participate to
that critical sequence of the cycle. I'm happy to report that the
engagement of developers around making Essex good is overwhelming.
Driven by technical leads that understand the challenges, significant
groups of old and new developers are participating, testing, assigning
themselves to bugs reported by others and fixing them in record time. As
project-focused groups, I think we are quickly maturing.
So, how far are we from Essex today ? The final release is set to April
5th, which means each core project needs to produce its final release
candidate before then. As of today no project did produce a release
candidate yet.
Swift is expected to release its final Essex version (1.4.8) on March
22. This version will be included in OpenStack Essex release, unless a
critical regression is detected in it.
Keystone (which every other project depends on) is still struggling with
9 RC bugs, including some key decisions to be made on configuration
handling. This is the hot spot right now: if you have free cycles,
please consider asking Joe Heck (heckj on IRC) how you can best help the
Keystone crew. I'd really like to have an RC1 for that project by
Thursday next week.
Glance also looks still a bit far away, with 5 RC bugs listed and slow
progress on them. I still hope Glance can be ready by Tuesday next week
though.
Nova is looking quite good, with 2 RC bugs left on the list. The trick
with Nova is that regressions and critical issues can hide in dark
corners of its 120k lines of code, so the focus is really on finding the
remaining issues, filing and targeting them. I expect we'll be able to
publish RC1 on Monday or Tuesday next week.
Horizon is almost ready (3 RC bugs left), waiting for a few fixes to
land in other projects before it can issue its RC1. It should be out
early next week.
Once all projects release their RC1, the hunt for the overlooked
release-critical issue will be on. It will be time to put the proposed
release to the test. In order to limit potential last-minute
regressions, only showstoppers will warrant a respin of the release
candidate, other bugs will be listed in the Known Bugs section of the
release notes. You can use the "essex-rc-potential" tag to mark bugs
that you think should be fixed before we release Essex.
Let's all make it rock !
Every 6 months, as a cycle ends and we prepare the next, we look back at
our release model and try to see how we can improve it. My opinion is
that we need (once more) to evolve it, and here is why.
Objectives and past evolutions
Our main objective is to produce stable and usable software. Our
secondary objective is let stable new features reach the hands of our
users in a timely manner. Our tertiary objective is to empower
developers to work efficiently on the features and improvements of
tomorrow. In simple solutions, those three objectives are not really
compatible, and in the past we tried to make adjustments without
acknowledging the fundamental incompatibility between those objectives.
The Austin-Cactus
system was a 3-month
cycle with various freezes. The issues with this system were that it
took too long to get features in the hands of testers and users,
developers had trouble working during the frozen periods, and not so
many people actually helped during the QA period.
For Diablo, we
decided to switch to a 6-month cycle with monthly milestones. Those
milestones were supposed to address the "get features into the hands of
users early" issue. To empower developers, we introduced an
almost-always-open trunk: we only had 2 weeks of feature freeze before
the Essex branch opened (and coexisted with the Diablo release branch).
The problem was the resulting quality was not good, since we accumulated
6 months' worth of features and only had 2 weeks' worth of QA on them.
So for Essex, there
was a decision that each project would decide where to place its feature
freeze and its first release candidate (which would be when the Folsom
branch opens). Most of them decided to use the essex-3 milestone as a
soft feature freeze and essex-4 as a hard one, which reintroduced the
"closed trunk" issue. And the work needed to stabilize 6 months' worth
of features is still daunting. And we saw lots of feature freeze
exceptions for last-minute "almost there" features that can't afford to
wait another 6 months.
So what's the solution ? Is there a good way to (at the same time)
reduce the pain of "missing a release", have always-open development
branches, and get higher quality ?
The kernel model
The only way to reduce the pain of missing a release is to have shorter
time-based releases. The only way to have always-open development
branches without sacrificing release quality is to separate them
completely from release branches. Where do those truisms lead us ?
Another famous open innovation project has already been there, and
that's the Linux kernel. Development happens in various always-open
topic branches, and regularly a merge window opens to propose stable
features for inclusion in the mainline kernel.
If we manage to efficiently separate OpenStack in topics, we could adopt
the same model. We could have several topic branches where core
developers on a specific area can collaborate and review code affecting
that area. We could have frequent releases (every 6-8 weeks ?), and for
each release we could have a "merge window" where stuff from topic
branches can be proposed for release, if deemed stable enough. Between
the moment the merge window closes and the moment the final release is
cut, various release candidates can be produced, on which serious QA can
be unleashed without blocking other developers.
This solves all objectives. If a feature is not ready yet (according to
the team maintaining the topic branch or according to release
management), it can bake until the next merge window, which is not far
away. Regular releases ensure that improvements reach the users in a
timely manner. Development branches are always open. And with a
reasonable amount of new code in every release, QA work is facilitated,
theoretically resulting in higher release quality.
Additional benefits
Splitting development into topics has an additional benefit: smaller
groups developing an expertise on a specific area make better reviewers
on that specific area than a random nova-core developer that can't be an
expert in all things Nova. I'd say that each topic should be small
enough to be manageable with a team of 6 core reviewers at the maximum.
Of course, nothing prevents anyone from being active on multiple topics
if they wish.
More frequent releases also allows you to set themes for each release.
It's difficult to refuse some feature in a 6-month cycle, but it's easy
to delay a feature to the next 6-week cycle. In the same way some kernel
releases introduce large architectural changes and some others are more
geared towards performance improvements or stability, we could also
define themes for every release -- after all the next one is not so far
away.
Challenges
There are a few issues with this model obviously. Code needs to be
componentized enough so that merge pains can be limited. Changes from
every release need to be merged back into topic branches. Code needs to
be clearly separated into a set of relevant topics, each with its own
set of core reviewers maintaining the topic branch. The release team
must be staffed enough to be able to review proposed code for stability.
Bug fixes need to easily end up in the release branch. And how do those
"releases" match the 6-month period between summits ? What means
"Folsom" in that respect ?
I hope we can use the following weeks to discuss the devil in the
details of this possible evolution, and be ready to take a decision when
the time comes for us to gather at the Folsom Design summit.
I'm back from Brussels, where happened the coldest FOSDEM ever. It
started on Friday night with the traditional beer event. Since the
Delirium was a bit small to host those thousands of frozen geeks, the
FOSDEM organizers had enlisted the whole block as approved bars !
On the Saturday, I spent most of my time in the Cloud and
Virtualization
devroom,
which I escaped only to see Simon Phipps announce the new
membership-based
OSI,
and Paolo Bonzini talking about the KVM ecosystem (in a not technical
enough way, IMO). My own OpenStack talk was made
a bit difficult due to the absence of mike to cover the 550-seat
Chavanne auditorium... but the next talks got one. The highlight of the
day was Ryan Lane's "infrastructure as an open source project"
presentation, about how Wikimedia Labs uses Git, Gerrit, Jenkins and
OpenStack to handle its infrastructure like a
contributor-driven open source project. The day ended with a good and
frank discussion between OpenStack developers,
with upstream projects and downstream distributions.
On Sunday I tried to hop between devrooms, but in a lot of cases the
room was full and I couldn't enter, so I spent more time in the hallway
track. I enjoyed Soren's talk about using more prediction algorithms
(instead of simple thresholds) in monitoring systems, introducing his
Surveilr
project.
The highlight of the day was Dan Berrangé's talk about using libvirt
to run sandboxed
applications,
using virt-sandbox. There are quite a few interesting uses for this, and
the performance penalty sounds more than acceptable.
Overall it was a great pleasure for me to attend FOSDEM this year.
Congratulations to the organizers again. I'll be back next year,
hopefully it will be warmer :)
In recent years, as open source becomes more ubiquitous, I've seen a new
breed of participants appearing. They push their code to GitHub like you
would wear a very visible good behavior marketing badge. They copy code
from multiple open source projects, modify it, but don't contribute back
their changes to upstream. They seem to consider open source as a trendy
all-you-can-eat buffet combined with a cool marketing gimmick.
In my opinion, this is not what open source is about. I see open source,
and more generally open innovation (which adds open design, open
development and open community), as a solution for the future. The world
is facing economical and ecological limits: it needs to stop designing
for obsolescence, produce smarter, reduce duplication of effort, and fix
the rift between consumers and producers. Open innovation encourages
synergy and collaboration. It reduces waste. It enables consumers to be
producers again. That's a noble goal, but without convergence, we can't
succeed.
The behavior of these new participants goes against that. I call this
the GitHub effect: you encourage access to the code, forking and
fragmentation, while you should encourage convergence and collaboration
on a key repository. And like having a "packaging made from recyclable
materials" sign on your product doesn't make it environment-friendly,
just publishing your own code somewhere under an open source license
doesn't really make it open.
On the extreme fringe of that movement, we also see the line with closed
source blurring. Building your own closed product on top of open source
technology, and/or abusing the word "Open" to imply that all you do is
open source, using the uncertainty to reap easy marketing benefits. I've
even seen a currently-closed-source project being featured as an open
source project to watch in 2012. We probably need to start playing
harder, denounce fake participants and celebrate good ones.
Some people tell me my view goes against making money with open source.
That might be true for easy, short-term money. But I don't think you
need to abuse open source to make money out of it. The long-term
benefits of open innovation are obvious, and like for green businesses,
good behavior and long-term profit go well together. Let's all make sure
we encourage collaboration and promote the good behavior, and hopefully
we'll fix this.
Next week, the European free and open source software developers will
converge to Brussels for FOSDEM. We took this
opportunity to apply for an OpenStack developers gathering in the
Virtualization and
Cloud
devroom.
At 6pm on Saturday (last session of the day), in the Chavanne room, we
will have a one-hour town hall meeting. If you're an existing OpenStack
contributor, a developer considering to join us, an upstream project
developer, a downstream distribution packager, or just curious about
OpenStack, you're welcome to join us ! I'll be there, Stefano Maffulli
(our community manager) will be there, and several OpenStack core
developers will be there.
We'll openly discuss issues and solutions about integration with
upstream projects, packaging, governance, development processes,
community or release cycles. In particular, we'll have a distribution
panel where every OpenStack distribution will be able to explain how
they support OpenStack and discuss what we can improve to make things
better for them.
And at the end of the session we can informally continue the discussion
around fine Belgian beers or their famous
Carbonade !
