Introduction -
Screaming fans, cheerleaders and streets blocked by the police greeted SoCal's Linux geeks last weekend. Admittedly the cops and frenzied crowds were there for the Grammy Awards and the cheerleaders were competing at whatever cheerleaders compete at, but the LUGs, commercial exhibitors, GNU/Linux and *BSD users and speakers couldn't help feeling special.
Just over 900 Free and Open Software users attended the exposition which presented 32 talks and 40 exhibition booths over the two days. Notable events included a presentation by Larry McVoy of BitMovers telling us that "BitKeeper is like crack!", well-attended tutorials on wireless and kernel internals, a GPG keysigning party and a controversial VoIP panel during which ISPs were accused of port-blocking, the issue of wiretapping was dismissed and foreign governments and ISPs accused of hostile actions. These talks and many more will be archived at the http://www.socallinuxexpo.com website.
Presentations/Talks
Perhaps the most maddening thing is to be presented with an embarrassment of choices. This was exactly the case for many of the talks as is inevitable in a multi-track environment. As pointed out by John 'Maddog' Hall in his talk, it's a tribute to the organizers that they managed to get so many good speakers. I managed to overcome my indecisiveness and sample the following:
Larry McVoy of BitKeeper fame (the content management system used by some of the Linux kernel developers, including Linus) gave an excellent presentation about "The Delta Development Model". This phrase is marketroid for a description of how the Linux kernel and some other FL/OSS projects are managed, with a central "dictator" figure and some lieutenants subsidiary to him, controlling the flow of patches.
LMcV set the stage for the need for BK-like software with a rough historical overview of the evolution of software development models. His tour led from the "Neanderthal" tarballing of single directories with no revision history, past the "Dark Ages" of RCS-like systems, past the "Middle Ages" of centralized version control exemplified by CVS and Subversion in which there is a division between those with commit powers (the ruling elites) and those on the fringes hacking on their bits of code (the peasants) and occasionally requesting the boon of a commit. Several disadvantages to this last model were pointed out by LMcV as aspects of a lack of scaling; the possibility of "breaking the tree"; the need to tag branches manually; and "merge hell" being chief among them. Finally, we ended up at the present with BitKeeper (and Darcs, monotone and Arch).
Developing his central theme of scaling, LMcV pointed out that initially Linus had examined each patch and made the decision to apply or discard it. As the kernel grew larger and more people sent patches things became a bit overwhelming, so Linus (working on the principle that this was asynchronous communication and there'd be a resend if it was important) would simply hold down the delete key until his mailbox reached a manageable size. As people grew wise to this they tried cron jobs to get his attention, but eventually that saturated too. Then it became obvious that there were some people that were trusted by Linus to send well-crafted, important patches for particular niches of the kernel. So, contributors sent their patches to those people instead, in the hope that they'd bring them to Linus' attention. Thus, the earlier described structure of a centralized body of committers with one "dictator" and several "lieutenants" emerged.
"Merge Hell" was experienced when those lieutenants submitted the accumulated patches, Linus took some of them (but not all!), then released a tarball of those changes. Meanwhile, new changes would have come in to the lieutenant and the evaluation of whether or not they were useful would have to begin all over again.
In "breaking the tree" it is possible for a committer to push unfinished code
into the tree in order to share work in progress with other developers.
Unfortunately, this will lead to wonky code being propagated to everyone, with
much tearing of hair and gnashing of teeth resulting. With the BitKeeper
system there is no single centralized repository. Instead there is a network
of distributed local repositories communicating in a P2P manner. The
client/server distinction has been abolished, with each local repository being
a full replica of the revision database. The logical unit that is propagated
through repositories is a "changeset" which spans files and is atomic.
Explaining how he had originally worked on a similar system for Sun Microsystems, Larry explained that he had got together with Linus and Dave Miller and Richard Henderson over a 4-hour session of drawing pictures on the floor to map out how they could alleviate the problems that developers had noticed before the adoption of BitKeeper. Specifically the evidence for how BitKeeper has benefitted the Linux kernel project is a measure of how much the kernel tree changed in terms of number of patches for 2 years on either side of the adoption of BitKeeper. This now stands at 50 patches per day, 365 days per year.
In summary LMcV pointed out that with changesets a developer could look at any line of code and in two mouse clicks view the patch that created it. He pointed out that the "diff3" merge algorithm took years to complete and they wouldn't be releasing details of it any time soon as it was their "secret magic sauce". He also noted, that renames were easy with BitKeeper, near impossible with CVS and difficult with Subversion.
Answering questions from the floor, LMcV said that he thought that although some of the "Open Source" projects like Darcs, Arch were getting close to providing this type of functionality, but that in practice they were way too slow, which made them unusable for a huge project like the Linux kernel (which contains around 60,000 patches/changesets). A final question brought out the potential problem of a conflict in the namespace for all the changesets held in all the individual repositories. LMcV admitted the theoretical possibility, but pointed out that with 50,000 BK users over the past 7 years there was not a single instance of this.
Randy Dunlap gave an overview of the work carried out by the KernelJanitors
project. This project was initiated by a Connectiva employee "Arnaldo
Corvehlo de Melo (ACME)" as a mentoring project for future kernel
contributors.
Giving an overview of the history of the maintainers and contributors to the project, Randy highlighted the diversity and openness, pointing out that some contributors had just started high-school and others were medical professionals who somehow were able to devote time to reducing function stack sizes, removing references to discarded code sections and other problems revealed by running a battery of static source code analyzers and other techniques.
ACME had started the project with a TODO list which consisted of low-priority
tasks that were accumulating, but needed to be cleaned up. This is still the
basis of the project and is an excellent starting point for anyone interested
in dipping their toes into the sea of kernel development.
http://janitor.kernelnewbies.org/TODO
An IRC channel, mailing lists, collection of resources (including Arjan van der Ven's "How NOT to write a device driver", and more importantly a community of folks like Randy who are willing to give feedback until you get it right are the center of this project.
RD also pointed out that his employer Open Source Development Labs (OSDL) provides access to their server test suite, which is available for those with an interest in seeing how well their kernel code runs including scaling issues on multi-cpu systems that might otherwise not be available.
http://www.osdl.org/stp
Norman gave a taste of his 4 day course dealing with development for Linux.
His very clear presentation style took the audience through the bootstrapping
process, showing how simple print statements could provide the user with an
idea of which stage the boot process was in and how the kernel could be compiled
with the kernel debugger enabled in order to provide more information. This left me wanting a further taste of his course, which is available through his training company. Unfortunately the shortness of the format meant that we couldn't cover as much ground as the title promised -- but realistically who could? At the end of his talk a largish crowd mobbed the podium seeking more information.
http://www.servin.com/LinDDProg.html
Marc Hamilton of Sun pitched the case for how Open is good for Sun and it's good for everyone. His presentation focussed on how it benefitted Sun to have a variety of choices of databases, middleware, etc. and used the example of how NFS has also benefitted by being an Open _standard_, resulting in the proliferation of an environment where Sun can flourish. He drew the audiences attention to the existence of http://www.java.net as an example of how a collaborative community is being encouraged by Sun Microsystems. He also pointed out that Sun Microsystem's recent release of over 1,500 patents is a more substantial gesture of good faith than that of IBM's recent 500 which includes expiring and irrelevant patents.
Questions from the audience about Sun's relationship with Microsoft drew a cautious response, as did questioning (from this reporter) as to why there was a need to not use the GPL if the CDDL was a wonderful for us as it was cracked up to be: the answer to that one was that patents were a probable concern, but that Marc was not qualified to speak about such legal issues.
Maddog gave a hugely entertaining talk about how the world of software and the globalized world were changing each other. One of his central points was that the production of software was moving back towards the small, or single programmer shop that it originally came from. The production of a highly capitalized piece of one-size-fits-all code by monolithic software houses is no longer a viable option.
Drawing upon the example of how localization of software is not happening (especially for users in the "third world" who have a plethora of languages and dialects, each of which has only a small number of users in global terms) because the return on the investment in proprietary systems doesn't merit it, Maddog pointed out that Open Source software can be fixed by these people, for themselves.
A further economic incentive driving this likely adoption of Open Source by developing countries is the massive income differential but constant unit pricing ( a legitimate, non-copied version of Microsoft's latest can be roughly the equivalent of 10 months wages in China ) of proprietary software. Open Source software on the other hand has no such disadvantage. (An interesting aside here was that in Thailand a survey showed that machines loaded with GNU/Linux were less likely to contain "pirated" software at a later date, than machines running Microsoft operating systems. This gives the lie to one of the myths perpetrated about GNU/Linux machines being used in developing countries).
Maddog was optimistic about the spread of Open Source as a result of these demands for l10n and customization and sees a probable move away from the selling of software as a product, to the selling of the service of _customization_ of software. In effect, a return to the small contracting shops of yesteryear.
The VoIP panel started rather quietly with the moderator (Gareth Greenaway, SCLUG) posing a series of questions to the panel selected by the organizing committee in advance.
The panel outlined the rosy future of not merely voice over IP, but integration of other types of data within this framework, and the interesting business opportunities waiting in the field. This section of the panel wasn't that informative or interesting, as the panellists were essentially pitching their business. A general consensus was evident that the traditional one-to-one mapping of a phone number to a particular geographic location was rapidly eroding due to cellphones and number transfers and that this would be developed and exploited by the nascent VoIP industry.
The problem of vested interests in both the phone companies and in national governments was discussed. Some of the panellists claimed that telcos were engaged in the practice of disrupting VoIP by port-blocking on a semi-random basis. Jeff Bonforte of SIPphone drew upon his previous experience of the Swiss government being closely tied to a single state phone company and thus having a vested interest in defending this phone company against new competition. Several questions from the audience as to whether or not it was likely that the large, existing telcos would ever actually give up their market dominance led to some diffused agreement among the panellists that it was necessary for there to be something done about this.
Things heated up when the floor opened to audience questions, with a slightly combative air being assumed by some panel and audience members.
The first substantive question was raised about QoS by an audience member who pointed out that it wasn't just about _outgoing_ QoS, but also _incoming_ and that ISPs could be non-cooperative about this if they chose. Both Louie Mamakos (Vonage) and Jeff Bonforte (SIPphone) dismissed this point, saying that it was mostly a red-herring. Jeff Bonforte asserted that it was necessary at times to introduce artificial delays into their international calls because otherwise the immediacy of the response was disturbing to customers used to a delay. Louie Mamakos pointed out that if the questioner was having a problem with this that he could always buy more bandwidth and Jeff Bonforte added that his customers were receiving CD quality sound. In addition to this Darryl Strauss added that he only kept a landline for some necessary application like DirecTV. Confusing the matter somewhat, one of the panellists then claimed that call-quality did not matter to customers anyway, as evidenced by the prevalence of cell-phones, with their frequently poor signals and dropped calls.
Now that the crowd was smelling blood another audience member asked whether or not these vendors would co-operate with foreign national governments to provide wiretap access. This appeared to agitate the panel members extremely and drew vocal reactions from most of the audience. Jeff Bonforte answered that they were a US corporation answerable to no-one but the US government and that their customers were buying equipment shipped from a US address, so why would they need to co-operate with a foreign government? The questioner then proceeded to point out that if the foreign government were a member of the WTO then they would possibly be obliged to co-operate. Darryl Strauss appeared to agree that international treaties might put a crimp in this feisty stance. There was some intimation that we all had the possibility of encryption anyway, and that moving to VoIP didn't make the situation any worse or better. The panel appeared keen to reassure the audience that this perceived problem did not matter, with Jeff Bonforte noting that at one of his previous gigs they had received frequent law-enforcement requests and although they had attempted to comply with them it was hard to get the logs and to track down the specific information requested. The result he intimated was that the requesters frequently gave up. Interestingly, given the obvious interest of the crowd in this topic and the reflexively defensive reaction of the panel, it turns out that this was one of the pre-tabled questions which the panel had agreed not to discuss because "it wasn't interesting"!
A final question from the audience about the 911 emergency call problem elicited the response that instead of fixating upon the absence of geographic-to-number mapping one should think about the new possibilities offered, such as the delivery of images or video back and forth between new devices and the caller.
Exhibitors
Perhaps the most visually impressive project was displayed by FlightGear who had constructed a very professionally finished mockup of a cockpit with a large projected rear display. Publicly available data was used ( from the sister TerraGear project http://www.terragear.com ) to create beautiful 3D scenery. While I watched someone flew around the Golden Gate bridge and returning later there was still a captivated audience watching as an airport modelled from FAA data was approached. I enjoyed this booth a lot, (even if some of them did apparently harbor the jocular remnants of bitter memories of the great "Red Hat gcc-2.96 Debacle" and directed me towards a copy of SuSE for rpms of FlightGear when they spotted the FedoraNEWS badge !). The project is fully GPL'ed and has a multitude of spin-offs in university research, training and games. In short a tribute to the vibrancy of Open Source and publicly available data.Finale
After a final look around the exhibitor booths, I left from the one exit not sealed off by the LAPD and into a warm Los Angeles afternoon, my ears ringing with the shrieks of bystanders as yet another black limousine crawled past with the windows half-down to expose the stars inside to their fans. A couple of lasting impressions stayed with me: the nearly complete absence of the word "free" from the presentations and exhibitors (apart from the FreeBSD and FSF booths), the enthusiasm of the audience in some of the more hands-on style presentations, and the excellent job the SCALE organizers did in bringing all this together.
FN Site Search
FN Blog Login