I've been asked multiple times why open source project management does
not fully adopt agile methodologies, which are so great. Or what are the
main differences between the two.
Agile is good for you
So first of all, I'd like to say that I think Agile methodologies are
great. Their primary value to me is to allow software development groups
to handle their stakeholders requirements in a sane way. By involving
developers more in the center of game, they contribute to use Autonomy
(one of the three main intrinsic motivators that Dan Pink mentions in
his book Drive) as a way to maximize a development team productivity.
Agile vs. Open
That said, applying pure Agile methods doesn't really work well for open
source project management. Some great concepts can be reused, like
frequent time-based releases, peer review, or test-driven development.
But most of the group tools assume a local, relatively small team. Doing
a morning stand-up meeting with a team of 60 in widely different
timezones is a bit difficult. It also assumes that project management
has some direct control over the developers (they can pick in the
backlog, but not outside), while there is no such thing in an open
development project.
The goals are also different. The main goal of Agile in my opinion is to
maximize a development team productivity. Optimizing the team velocity
so that the most can be achieved by a given-size team. The main goal of
Open source project management is not to maximize productivity. It's to
maximize contributions. Produce the most, with the largest group of
people possible.
That's why open source project management is all about setting the
framework and the rules of play (what can get in trunk and how), and
about trying to keep track of what is being done (to minimize confusion
and friction between groups of developers). That's why our release
cycles are slightly longer than Agile sprints, to have a cadence that is
more inclusive of development styles, and to enforce time to focus, as a
group, on QA before a release.
Agile devs in Open source
It's difficult for Agile developers to abandon their nice tools and
adopt seemingly-more-confusing open source bazaar ways. But in the end,
I think open source is more empowering, by addressing the two other Dan
Pink types of intrinsic motivators, Purpose and Mastery. Working on
an open source project and contributing to the world's amount of public
knowledge obviously gives an individual a sense of purpose to his work,
but even more important is mastery.
Each developer in an open source project actually represents himself.
With all proceedings and production being public, in the end his
personal name is attached to it. He builds mastery and influence over
the project by his own actions, not by the name of the company that pays
his bills. Of course his employer has requirements and usually pays him
to work on something specific, but the developer acts as the gateway to
get his employer's requirements into the open source project. That way
of handling stakeholders requirements places individual developers at
the very center of the game, even more than Agile does. You end up with
the highest number of highly-motivated individuals, which in turn leads
to lots of stuff getting done.
Agile subteams
Finally, nothing prevents an open source project to have Agile
development subgroups contributing to it. These subgroups can have user
stories, planning poker, feature backlogs, pair programming and stand-up
meetings. There are multiple challenges though. Aligning agile sprints
with the open source project's common development schedule is tricky.
The Agile work schedule needs to be adapted to make room for generic
open source project tasks like random code reviews or pre-release QA.
Some other group may end up implementing a feature from your internal
backlog, and communicating the backlog outside the group can be
bothersome and challenging.
I'd like to find ways, though. What do you think ? Can Agile and Open
live in harmony ? Should they try ?
The OpenStack Bexar release now
passed FeatureFreeze.
Branches containing needed new features or changing the expected
behavior of the software can still require an exception and be proposed,
but their benefit will have to outweigh the regression risks they bring
for them to be part of the Bexar release.
Some insight on how well we managed to hit the objectives of this
deadline (% of specs that were merged in time for the freeze):
- Essential specs: 3 merged (100%)
- High-prio specs: 14 merged, 1 still proposed (93%)
- Medium-prio specs: 9 merged, 2 still proposed, 4 deferred (60%)
- Low-prio specs: 8 merged, 2 still proposed, 2 deferred (66%)
So we did a great job of getting High-prio proposed branches merged.
Thanks to a final push on the freeze day, we also managed to get most of
the Medium and Low priority branches in. A few previously-untargeted
branches made it, like switching to CoW format by
default,
support for availability
zones
or CEPH
volumes,
for a total of 45 targeted
specs. The overall hit score
is just above 75%, which is amazing for that number of objectives.
Next stop is in 12
days, GammaFreeze (Jan 25).
Until then, we need to get as many bugfixes in as possible. Now that
(most) feature branches have landed, it's time to put your QA suit on
and test, report and fix all issues you encounter. Let's make Bexar a
great release !
The OpenStack Bexar release now passed
BranchMergeProposalFreeze.
Branches containing new features or changing the behavior of the
software can still require an exception and be proposed, but there is no
guarantee they will be accepted and be part of the Bexar release.
Some insight on how well we managed to hit the objectives of this
deadline (% of specs that were proposed in time for the freeze):
- Essential specs: 3 merged (100%)
- High-prio specs: 8 merged, 5-6 proposed, 2-1 late (87-93%)
- Medium-prio specs: 5 merged, 4 proposed, 3 late, 3 deferred (60%)
- Low-prio specs: 2 merged, 4 proposed, 3 late (66%)
Given the very high number of specs targeted to Bexar, this is quite
good (I was expecting something around 100%, 75%, 50% and 25%). The
overall score is around 75%, which is amazing with 42 targeted
specs in 3 months. Congrats
to all the developers ! We also had a few unexpected specs that made it,
we will retrospectively add them to the picture.
Next stop is next week,
FeatureFreeze (Jan 13): all
feature branches need to be merged so that we can safely switch to
QA/testing/bugfix gear, 3 weeks away from release. Given how well the
reviews go, I hope we will be able to sneak a few late branches in and
further improve the scores.
OpenStack is busy with so much development activity it's hard to keep
up. 42 (!) specs were
targeted for the 3-month long Bexar development
cycle... and there are
more than 150 active branches. Over the last month alone, we saw 750
commits by 50 different people. Taking a step back, what new features
should you expect to land on February 3rd, in the Bexar release ?
Swift (OpenStack object storage)
The big news in Swift is support for unlimited object size, through the
implementation of client-side
chunking.
The only size limit for your objects is now the available size in your
Swift cluster ! You can read more about that exciting feature in John
Dickinson's blog
post.
We also hope to ship
Swauth,
DevAuth highly scalable replacement, directly into Swift codebase.
Exposure of most of the S3 API in
Swift may or
may not make it.
Glance (OpenStack image registry and delivery service)
The Glance image service will expose a unified REST
API (no more
distinction between the image registry and the image delivery services).
We will also have the possibility to upload image data and metadata over
one single
call.
Unified client
classes will be
shipped directly in Glance. We also hope to have a S3
backend...
Nova (OpenStack compute)
There is so much coming up in Nova it's hard to summarize. Nova will
make use of those new Glance client
classes,
obviously. We will support booting VMs from raw disk
images
(rather than a kernel/ramdisk/image combination) and have a rescue
mode to mount
your faulty disks under a sane environment. We plan to have
instance
snapshots
ready. API servers can now
expose
optional admin features (through the --allow_admin_api flag), like a
specific XenServer instance
pause or
suspend
feature.
Lots of improvements might go unnoticed, like the
internationalization
of messages, the standardization on services using eventlet, more robust
logging,
or the move of the IP allocation down the
stack.
We'll also finalize some incomplete features, like access to your
project VLAN through a
VPN, security
groups
that work in all network modes, and
Hyper-V
support.
We hope to have much more: a web-based serial
console
to access your VMs,
ipv6
support, the possibility to deploy hardware in a staging
area of
your cloud, support for highly available block volumes through
Sheepdog,
instance
diagnostics
allowing to retrieve a history of actions on instances, the possibility
to do live
migration
in nova-manage, iSCSI
support
for XenAPI... But let's be realistic, not everything will land in time.
What doesn't make it will certainly be in the next release, Cactus,
which will be released in April !
Congrats to our awesome development team for making all this possible.
Those last two months have been a very fun ride for me :)
In his book Where in the world is my team, Terence Brake outlines the
three challenges that global and virtual teams face. There is
Isolation (reduced contacts and difficulty of trust building can
easily make you feel alone and lose motivation), Fragmentation
(unclear purpose and fuzzy responsabilities fragment your effort and
make you inefficient) and Confusion (too much or too little
information makes you take the wrong path, and hidden activities prevent
anyone from noticing).
As virtual and global aggregations of common interests, community-driven
open source projects are specifically vulnerable to those challenges.
Fortunately, Brake also explains how to fight these. One of the areas he
identified is convergence, through shared understandings, as a way to
generate clarity and fight confusion (and, to a lesser extent,
fragmentation).
Without shared understandings, the lack of shared context, the
conflicting assumptions, and the distance between team members can
generate a lot of confusion. This confusion results in the loss of
everyone's time in the best case, in team implosion in the worst. With
shared understandings, you get natural convergence and you reduce the
need for interrupt-driven communication.
That's why we, the OpenStack team, need documentation. Not only
user-oriented and developer-oriented documentation, but also project
documentation. The team already has a clear
mission. Meeting face-to-face
during our design summits allows us to get to know each other, which is
critical to bootstrap common context. We need to ensure we have team
principles (we have design
tenets and coding
standards, we need a code of
conduct), clear priorities, open implementation plans, shared
performance indicators... This won't happen in a day.
Since I joined the project, I tried to produce a basic set of reference
documents which should help generating clarity. So far I concentrated on
our release cycle and our Launchpad tools:
Sometimes it looks like those project documentation pages are spelling
the obvious, but that's the price to pay to make sure everyone doesn't
have different assumptions. Those wiki pages are very much open for
discussion and evolution: the goal is not to force anyone into new
workflows or extra bureaucracy. The goal is to have a clear reference
point when you need more clarity. I hope it will help avoiding confusion
by establishing shared understandings.
