Wouldn't it be nice ...

People were so busy complaining about my pie chart that most of them apparently didn't have a chance to think about the meaning of the actual data. To try helping people look at the information rather than its presentation, here's a bar chart of the same information:

I don't recommend looking at it because you may go blind, but I've made available the (extremely ugly) script that created this.

I ran a quick, informal poll on the internal Gentoo developers' list last week, and tonight I began analyzing the results. 50 developers responded to my 9-question survey, and I'm going to post the results of 1 question at a time.

First question: What are the top 3 issues facing Gentoo?



Technical issues are way down on the list. Developers' top 5 issues are manpower, publicity, goals, developer friction, and leadership. It's good to see that we've been addressing at least a couple of them with the newly energized public relations project and work on the Code of Conduct. Other issues that have been ongoing for quite a while now are the lack of distro-wide vision and goals. The Council could provide those by increased activity and taking stronger stands in particular directions, and that's part of the reason I did this survey—to figure out which directions our developers care about. I think part of the problem is that nobody sits around pondering directions and ideas. Everybody's busy working in their own little areas and not thinking about the big picture. Manpower, or lack of it, is another issue I'm indirectly addressing in my push for greatness, which I'm going to post more about at some point (I promise!).

To create this chart, I used Google's excellent chart API. The neat part about the API is that it's simply a URL, so you can construct it with any language. I used a shell script since I was already fiddling around with awk. Any answer with less than 4 respondants was grouped into Other to make the rest of the chart readable.

That's a question a lot of people have been asking lately, with the news about the nonprofit foundation, the lack of news updates on the homepage, and the canceled release. I answer it in a short LWN article.

I've been talking with Lance lately about setting up a good development machine, and GCC 4.3 (still unreleased) came up so I checked out its changes. Here's a few I found particularly cool:

• A new forward propagation pass on RTL was added. The new pass replaces several slower transformations, resulting in compile-time improvements as well as better code generation in some cases.


• A new command-line switch -frecord-gcc-switches has been added to GCC, although it is only enabled for some targets. The switch causes the command line that was used to invoke the compiler to be recorded into the object file that is being created. The exact format of this recording is target and binary file format dependent, but it usually takes the form of a note section containing ASCII text. The switch is related to the -fverbose-asm switch, but that one only records the information in the assembler output file as comments, so the information never reaches the object file.


• A new internal representation for GIMPLE statements has been contributed, resulting in compile-time memory savings.


• A new command-line option -fdirectives-only has been added. It enables a special preprocessing mode which improves the performance of applications like distcc and ccache.


• Experimental support for the upcoming ISO C++ standard, C++0x


• Fortran: The -fexternal-blas option has been added, which generates calls to BLAS routines for intrinsic matrix operations such as matmul rather than using the built-in algorithms.


• Fortran: Support to give a backtrace (compiler flag -fbacktrace or environment variable GFORTRAN_ERROR_BACKTRACE; on glibc systems only) or a core dump (-fdump-core, GFORTRAN_ERROR_DUMPCORE) when a run-time error occurred.


• Java: libgcj now supports all 1.5 language features which require runtime support: foreach, enum, annotations, generics, and auto-boxing.


• x86/amd64: Tuning for Intel Core 2 processors is available via -mtune=core2 and -march=core2.


• x86/amd64: Code generation of block move (memcpy) and block set (memset) was rewritten. GCC can now pick the best algorithm (loop, unrolled loop, instruction with rep prefix or a library call) based on the size of the block being copied and the CPU being optimized for. A new option -minline-stringops-dynamically has been added. With this option string operations of unknown size are expanded such that small blocks are copied by in-line code, while for large blocks a library call is used. This results in faster code than -minline-all-stringops when the library implementation is capable of using cache hierarchy hints.


• x86/amd64: Support for SSSE3 built-in functions and code generation are available via -mssse3.


• x86/amd64: Both SSE4.1 and SSE4.2 support can be enabled via -msse4.


• x86/amd64: GCC can now utilize the ACML library for vectorizing calls to a set of C99 functions on x86_64 if -mveclibabi=acml is specified and you link to an ACML ABI compatible library.


• MIPS: libffi and libjava now support all three GNU/Linux ABIs: o32, n32 and n64. Every GNU/Linux configuration now builds these libraries by default.


• The configure options --with-pkgversion and --with-bugurl have been added. These allow distributors of GCC to include a distributor-specific string in manuals and --version output and to specify the URL for reporting bugs in their versions of GCC.

I'm already using GCC 4.2, but I hadn't really looked into its changes either until now:

• OpenMP is now supported for the C, C++ and Fortran compilers.


• A new command-line option -Waddress has been added to warn about suspicious uses of memory addresses as, for example, using the address of a function in a conditional expression, and comparisons against the memory address of a string literal. This warning is enabled by -Wall.


• C++: -Wextra will produce warnings for if statements with a semicolon as the only body


• C++/libstdc++: Enabled library-wide visibility control, allowing -fvisibility to be used.


• x86/amd64: -mtune=generic can now be used to generate code running well on common x86 chips. This includes AMD Athlon, AMD Opteron, Intel Pentium-M, Intel Pentium 4 and Intel Core 2.


• x86/amd64: -mtune=native and -march=native will produce code optimized for the host architecture as detected using the cpuid instruction.

LWN just published my story of this name as premium content. I ran a number of analyses on Gentoo's CVS commit history, kindly provided by Robin Johnson, to look at what happened to our developers and our commits over time. If you want to know the results now (rather than waiting until the content becomes free), subscribe to LWN—the best Linux and open-source software news.

I recently posted about making Gentoo a better tool. A requirement for being a good tool is being a tool that doesn't break—thus, we need to improve our quality to a more reliable level. I'm going to mention a few ideas to start this discussion, which I hope the rest of our community will participate in.

First, we essentially have no code review. About the only time any code in Gentoo is reviewed is before and during a developer's training, with a notable exception being the requirement to post eclasses to the gentoo-dev list. Increasing our code review ought to result in an increase in quality, in ability to justify code in words, and in a stronger community of contributors.

How do we increase code review? One idea is to require reviewer approval prior to committing, but this isn't the best answer for Gentoo. We've always been a pretty open community. Developers aren't prohibited by ACL from committing anywhere in our ebuild repository, so I don't think they would accept additional requirements that increased the burden of contributing.

Instead, let's create a gentoo-commits mailing list or RSS feed(s), with full diffs. We should use this tool in many different ways.

• Each team should use it internally to review all commits to its packages.


• Mentors should continue to follow their mentees' commits well after they're granted commit access (6 months minimum, and I recommend forever).


• Mentees should also review their mentors' commits, first to learn and later to review.


• Every developer should have at least one reviewer and review at least one other developer. This should be formal and documented to ensure it's happening.

These uses will require that the commit diffs be easily filterable by both committer and files affected. RSS feeds could be made available based on developer or herd, and e-mail lists could contain the information in e-mail subjects or headers.

Second, we should improve our unit testing, where the units are individual packages. This can be both automated and performed by developers and arch testers. Although a number of packages have a useful, working test suite, most lack one. For these packages, we should attempt to provide something automatable in src_test() even when a test suite is absent. Failing that, we should print out a checklist in src_test() of tests to perform before stabilizing a package. There should never be an empty src_test().

Another package-level testing approach is to create solid, automated tinderboxes. This remains unrealistic for our entire database of 10,000+ packages, but we should at least get this going for our "system" set and perhaps for some of the most common sets of packages for servers and desktops. Exactly how to set this up remains a question, since there's a lot of tinderbox code floating around. Bonsaikitten has some almost-working code based on swegener's work; Catalyst has some tinderboxing capability; or we could look into using Mozilla's tinderbox.

Third, we should improve our integration testing, on the entire repository level. Our main source of testing here will be our users, because they have infinitely more combinations of build options and hardware than we can reproduce on Gentoo infrastructure. But how can we take advantage of this testing to improve our quality? By creating an additional, time-lagged set of rsync mirrors with additional QA checks, we could allow users who want to test the latest and greatest software to help those who want stable and solid software.

We already have keywords for ~arch and arch, but they're still too mixed. A problem in ~arch ebuilds can break the entire tree for all users. They really need a stronger separation. Perhaps the separate repositories should be ~arch versus stable. But another way to do it is to add a delay to the second set of repos, anywhere from 24 hours to a week. This delay allows us time to encounter major problems in the fast-sync repos, fix them, and carry the fixes over to the slow-sync repo. But we'll need a way to make this really easy to do. It feels like branching with periodic merges, along with cherry picks of major bugfixes, is the right way to do this. Unfortunately, CVS sucks at this. We may need to migrate to a more capable version-control system before this option becomes realistic. In addition to the user testing, we could add a tinderbox into the slow-sync repos to require that they build with the most common configurations.

To sum up, I want to increase code review, unit testing, and integration testing. These three things will strengthen Gentoo's quality, reputation, and community.

CIA commit stats: 328 messages so far today

I'm giving a short talk at OSCON next week on Gentoo's progress in the past year. So please comment on my blog to let me know areas you want me to cover. I'm open to any and all ideas.

Thanks!

BTW: Anybody living near Portland got a couch I could crash on Tuesday, Wednesday, or Thursday of next week (July 24-26, 2007)?

Gentoo is a metadistribution, meaning you're supposed to built whatever you want with it. And we provide some of the tools to make that a reality, but we stop short. We still make it too hard to do what you want with Gentoo. Here's some of the ideas I want to explore over the next year:

• Increased binary package support: We already have an experimental tinderbox setup to build packages for a variety of architectures. I want to investigate increasing our collection of binary packages to make a fairly complete installation possible using only binaries.


• Repositories for user-created distributions: We have tools like Catalyst to create basic installations with stage3 or stage4 tarballs, but it's still hard to get what you want with them. We should make a repository of Catalyst spec files for a variety of purposes, and we should also add an accompanying mirror of the created tarballs so anyone can use them. We could also revive the Seeds project to create not just vanilla stage4 tarballs but fully fledged, preconfigured installations customized for specific purposes like LAMP servers, development machines, various cluster nodes, GNOME/KDE/Xfce desktops, etc. We already have the possibilities; we need to share our tools.


• Creating the Gentoo metacommunity: In addition to letting you create your own distribution from Gentoo, I want Gentoo's community to be what you want it to be. We're in the middle of adding a few new mailing lists because our primary development list is drowning in noise. We made gentoo-project for non-technical general talk and gentoo-dev-announce for development-related information so developers don't need to slog through gentoo-dev. I want to take that a step further by forming stronger Gentoo microcommunities around specific areas and moving discussion about those areas to their IRC channels and mailing lists. Our development community has grown too big to keep everything on a general development list. Everyone tries to chip into every discussion, even if they have no relation to it or are unaffected by it.


• Excellence: Gentoo's QA is not the greatest. What can we do to improve this? Some automated tools exist for pre-commit checking; can we add anything server-side? Can we add some build servers for critical system packages, so they don't make it to users before building in a predefined set of common configurations? Can we improve our developer recruitment or add new training for existing developers? How can we renew and strengthen our commitment to excellence? You can't create a tool with Gentoo if it's broken.


• New tools for new places: This is more of an exploratory idea. I'd like a team of contributors to research all the places people use Gentoo, and put together a collection of tools we're missing. Further, we need to do more extensive research on other distributions to see what sorts of tools they have that we lack. I have no symptoms of NIH syndrome, so I'm happy to pull in working tools written for other distributions; I already did this for many of Fedora's system-config-* tools last year.


• Resolving the meta vs. specificity conflict: Gentoo's status as a metadistribution sometimes produces conflicts between the goals of various projects and teams within Gentoo. Sometimes a person simply needs to make a decision of how to deal with it: pick one, or find a way to do both? Many people remarked over the past couple of years that we're lacking strong leadership and overarching goals, which would give us some consistent way to decide which projects will "win." As Seemant has said, Gentoo is a framework, a metadistribution, not a place that forces people to go a certain way. We should strive to enable contributors to follow whichever path beckons to them. But where is Gentoo going, and where should it go? You decide. The innovators and leaders can only suggest places for it to go, so the rest of the community has to follow them to our destination.

Haskell: How and Why?
Bart Massey, Portland State U.

Functional programming language
- General purpose

The next big thing? Driven by open source adoption

Modularity

You give up global variables, and variables altogether.

Incrementally compiled, like Python bytecode

And now I have to leave to go home...

I'm going to post my notes from the BarCamp Portland sessions here.

Packed room. We started out brainstorming one word each that's related to the user experience. Ideas include: results, easy, story, predictable, relevant.

Startups: Don't be afraid to throw some UI paradigm away. Your users may hate it, however much you love it.

Does gradually changing the UI work, or piss people off? No consensus.

Techies are really bad at pretending to be the naive users, so test it on real users!

Accessible interfaces as secondary UIs are different and need to be tested as such.

User personas are caricatures of reality and don't model true users well.

And then we ran out of time...

I just posted this to the Gentoo development list, but I thought other projects could use a similar idea so I decided to add it to my blog.

Hi all,

I'm sure I'm not the only one with a number of projects I'll never get to, but I'd really like them to happen anyway. I suggest we create some sort of page that aggregates all of these personal projects together, so anyone can browse through them and look for stuff that sounds fun.

The goal is to increase contributions from outside by giving them a ready list of projects of all sizes and difficulty levels to work on, projects that go beyond what happens at Bugday. Further, it could also help current Gentoo developers who are bored or have lost interest in what they're doing by helping them to find somewhere new to contribute.

A prototype with just my projects is at http://dev.gentoo.org/~dberkholz/proj/

Thanks for your comments!
Donnie

I just came across another good post on why it's worthwhile to use Gentoo, if not permanently, at least temporarily. Kathy Sierra posted on the Passionate Users blog about the inverse relationship between efficiency of your tool and your understanding of what it does. The more autoconfiguration there is, the harder it is for you to figure out what to do when you've got a setup that doesn't autoconfigure.

A commenter on that post pointed to an earlier Joel Spolsky (Joel on Software) post on leaky abstractions—in other words, an incomplete, simplified metaphor that sometimes breaks down. When you don't understand both sides of the metaphor, you can't figure out why everything's busted.

In the same way, automated tools suck. They're great once you understand the black box they hide from you, but ignoring that black box entirely is a route to disaster. That's why you should try Gentoo, or LFS, or some similar DIY distro that gives you the opportunity to understand what all those fancy GUI tools really do. Someday, they'll break. Will you survive, or will you take your toys and go home?

The deadline for proposing ideas for OSCON talks was Monday at 11:59 p.m. Naturally I clicked Submit for the final time not long before then, at 11:56 p.m. Note I said "final time," implying more than one—I decided to go all-out and submit three ideas, in hopes that at least one will slip through. For the curious, they are

• Design and publish beautifully and professionally: How to create a professional-looking document, using Scribus and Inkscape. Many people who read this probably don't realize I spent four years working as a page designer and copy editor at newspapers. I continue that interest today as editor of a newsletter about open-source activity for the Open Source Educational Laboratory at Oregon State.


• Community dynamics in a large open-source project: Problems, solutions and conundrums in Gentoo. The funniest (and saddest) part is when the same things begin to repeat themselves, and nobody else remembers last time it happened.


• Open-source software in the biosciences: Where we are and what we learned on the voyage: The cruel joke of scientists as programmers. As a grad student in biochemistry at OSU, I've had to deal with more ugly software than many of you can imagine. But at least more of it is open source now, right?

Anyone else submit proposals? I'd be interested to hear what they're about.

I'm taking over a webapp at work written in PHP+MySQL. I'd like to add some Ajaxy goodness to it because the UI is an ugly POS with lots of pointless page loads for little things, etc. Funny thing is, I have basically no experience in any of these technologies.

Suggestions for the best way to learn them together or separately?

Thanks!

Thanks to genstef, I've finally moved my overlay hosting to overlays.gentoo.org. The new way to grab a copy is:

git clone http://overlays.gentoo.org/git/dev/dberkholz/

I'll update the other READMEs and such floating around shortly, and attempt to set up a cron job to sync over to the old location daily or so. Flameeyes, you're also a committer. Anyone else who would like commit access, let me know and give me a patch or two to stuff I care about.

For anyone who's interested, here are the projects I've got going right now. Many of them could use some help, so take a look and let me know if you're interested in any. Roughly in my order of interest:

• Add the Sugar desktop environment for OLPC — it's in my overlay, but it segfaults on startup of sugar-emulator somewhere in sugar-shell code. Try it out and see whether you can come up with a fix.


• Port LTSP to Gentoo — pioto, straaken and perhaps another person or two are working with me on this. This involves changes to the client-building plugins, init scripts, and adding some ebuilds. Also, probably creating Seeds for the client and server.


• Get the rest of the system-config-* GUI tools from Red Hat working — some remain masked. Would appreciate testing and fixing on any that remain masked.


• Add virt-manager into the main tree from my overlay — haven't got a Xen instance to test it with yet. If anyone would like to test this and let me know, that'd be great.


• Fix our X init scripts to be more like upstream intended, then fixing upstream to be current. havner is taking the lead on this, and I look forward to seeing his work.


• Add some new science packages, including KiNG and friends from the Richardson lab, CCP4MG, CCTBX and more.


• The infamous bug #44132 — make multiple MPI implementations simultaneously installable.


• Resume my occasional series of blog posts on Gentoo in the enterprise, embedded, cluster etc environments. One post I want to make is how to use the Gentoo installer's CLI frontend to make large, automated installations easy.

And of course these are beyond the usual ongoing maintenance of X, science packages and cluster packages.

I plan to write a few articles about how to use Gentoo in widely varying areas, from the enterprise to the development box. I'd appreciate some suggestions on good areas to pick, any tips about any of these areas that you use, or anything else relevant.