I think the webteam deserves some more credit for their work, so I’m going to highlight some of the remarkable things they pulled of the last months. The first ‘challenge’ they had was to give the new bugzilla a new layout. The new bugzilla was rushed from the sysadmin side, this was caused by a serious security issue we found. Hence the webteam only had a week to come up with some layout. And they did. Unfortunately a week was too little time, and at the launch the new layout was not so good. Then the team had to operate for a few weeks with a smaller team, but when the team was back on full strength, bugs.kde.org was fixed and it now rocks. I’m very proud at it.

Another project the webteam took on was to apply Neverland to forum.kde.org. The forum is a big site, with a lot of users. Unfortunately the forum is very underestimated by developers. It really is a fun place to hang around. Some developers now start to discover the forum and are very excited about it. It’s a great place to communicate in a pleasant way with actual users of your application. Much better than via bugs.kde.org in my opinion. If you want a forum for your application, just let the team know. Anyhow, they rolled out the new theme there too, which was a big chunk of work. If you want a *very* cool impression of the changes and about the possibilities of Neverland, just look at their Tour!

And more recently they applied the new Neverland theme to download.kde.org. Instead of a boring index of all files and folders, it now has a nice background, nice fonts and a nice expandable header with the contents of the .message file in each folder, all in line with the rest of the Neverland-sites.

So a big thanks to the webteam. Especially Sayak Banerjee, Eugene Trounev and Ingo Malchow!

We are looking for a virtual machine which we can use and which will be dedicated to provide an anonymous git service to our community. We had a couple of such servers until recently, but we had to say goodbye to one, because the SVN-size of our repo is also still growing, and that server did not have enough disk space to do both. Another one had some load issues, so we had to make a choice to take that one out of rotation too. We still have 2 left, but one malfunctioned today. And then suddenly we were left with only 1.

Hence this call for a new server. The specs are not that high, just some disk space (50GB would be nice), root access, somewhat good processors and a good internet connection.

We do have a small preference for a company this time, people offering a part of their private server is always appreciated, but for an anongit server, we would like good uptimes, quick response when there are problems, etc. Other wishes we have involve a KDE pastebin for images, some enhancements to files.kde.org and some more, so if we get more than one offer, I hope we can use the servers for those wishes.

In return you get list at our famous Thank You-page. We can even put your logo on there if the offer is very generous :) If you are interested mail us at sysadmin@kde.org.

download.kde.org is the last step for users before getting the actual stuff we have released. It’s a mirror network, we have a list of servers all around the world that are mirroring the software we release. After we publish a new tarball, the mirrors pick it up. Our task is then to redirect the user to a mirror and that way we share the load over a lot of servers.

In the old setup users had to manually select a mirror. We did try to help you by guessing the right country, but that did not work that well. In the new situation you will just see a directory overview on http://download.kde.org. But when you click on a file, it will select a mirror in your country or – if that does not exist – in your continent. Fully behind the scenes, you just get your file.

If you are on the same network as one of the servers, it will automatically pick that one. That will give you awesome download speed. Great if you are lucky enough to be on a university network which also mirrors KDE. You can find a list of our official mirrors here

When you press the detail button from the last column. You get to a page with more info. For example here. That holds a list of the mirrors, not a simple list. But it will only show the mirrors that are up and running and the ones that can actual serve the file.

On that page are meta-links. That’s very cool, because a download manager can deal with it. For example KGet. If clicked, KGet will start downloading the file from several mirrors at the same time. Each a section. The meta-data holds information about checksums for each section, so you are sure at the end you end up with the right complete file.

The page also holds a big download button for impatient, checksums for the control freaks and if we signed the file, it also shows the PGP-signature for the paranoids.

In summary the new download.kde.org is a drop in replacement, everything should just be fine. But application developers linking to files on download.kde.org might want to add “.mirrorlist” behind their current download url’s to provide a nicer download experience for users.

All this is made possible by MirrorBrain. An awesome piece of software, also powering openSuse, LibreOffice and many others.

ps. Another great way to contribute to KDE is to simply join the game

If you don’t reactivate your account before March 30th we will block your account from logging in until the password reset feature is used to set a new password.

Sorry for the trouble, but we hope you will enjoy the upgrade.

Currently bugs.kde.org runs bugzilla version 3.2.10. In the meanwhile upstream released 3.4, 3.6 and 4.0-series releases. And just a couple days ago there was a 4.2 release. It’s clear we ran behind, though 3.2 series still received security updates and we followed those.

But then we felt that we should upgrade to 4.2. Last Saturday we copied the current bugs.kde.org-site and performed the test upgrade. We discovered some issues during the upgrade. For example there are some Perl dependencies which can not be met on a Debian Squeeze installation. For the test we pulled them from CPAN, but that is not really acceptable for the final migration. We created Debian packages for everything missing, so that should be sorted now.

We were also able to test the custom modifications we made bugzilla. Christian Ruppert stepped up, and did most of the work of the porting, which was very friendly. These were the changes that blocked us in the past for the upgrades.

Another problem is the theme. The current theme needed porting to the new version. Ingo Malchow is now working fanatically to get this in shape before the final conversion this Friday. I’ve seen screenies, and that looks very good already.

The test conversion was also good to have some experience in the process and to get an idea how long the conversion will take. We have made an extensive plan, which we can follow step-by-step. This Friday the final conversion will happen. This means bugs.kde.org will be down while that is happening. We expect the downtime to be not less than 4 hours in total.

I want to stress that this upgrade is completely unrelated to a possible workflow change. Some people have taken my announcement as starting point to start a discussion about a workflow change. That’s fine but it will not happen this Friday, nor in the next couple of weeks to be honest. It would be good if the community reaches consensus though and give the sysadmin team a clear plan to execute. I suggest everyone to contribute to the thread on kde-core-devel.

Upgrading is a bit harder than usual, because KDE has made modifications to it. We have a wizard guiding people for example. That code probably needs some polishing to work on a newer version.

What we need is:

• someone who is not afraid of big databases
• someone with knowledge of perl
• someone who has experience with bugzilla and its templating engine.

That’s all :) Ideally we would like a group of people taking on this task, it’s a bit to much for one person probably. Of course the regular sysadmin team will provide support to facilitate the upgrade.

Please mail us at sysadmin@kde.org if you are interested! You know: Many hands make light work!

Ok, that is a bit harsh, but still. Release parties are organised all around the world to celebrate the release of KDE 4.8. You should go to one, most of them are relaxed, friendly and allows you to meet your local developers, contributors or just fellow KDE users.

There is also a Release Party planned in Culemborg, The Netherlands. It’s just south of Utrecht, a beautiful city in the center of the country. In the middle of the region known for the jam-production. The release party won’t have any jam though, we will stick to Pizza, cola, a soccer table, a presentation done by Sebastian Kügler and a demo of Plasma Active.

Entrance is free and everyone who wants to come is welcome. All you have to do is sign up.

During KDE’s (one of the largest open source communities around, with about 2500 active developer accounts with direct write access to many millions of lines of code across dozens of products, and large numbers of external contributors) ongoing migration from SVN to Git, GitHub was never considered as an option because the community considers it unacceptable for an open source community to throw their weight behind a proprietary solution.

Instead we seriously considered the open source alternative Gitorious.org for a while, but ultimately were unable to come to an agreement with Shortcut AS with regard to a satisfactory hosting plan, and also had troubles making its ACL system work for our needs in some exploratory migrations to the platform.

Self-hosting the Gitorious software also fell out of the running fairly quickly due to the ACL problems and other concerns about the software’s features and implementation; while KDE probably would have been able to put together a squad of coders to work on the software in its interest, Shortcut at the time had failed to get a dev community around Gitorious off the ground and was unacceptably slow at processing merge requests – meaning a fork might have become necessary, which is always an icky proposition not to mention a big burden on resources.

Ultimately we ended up banging together our own Git infrastructure out of several smaller components, some of which we wrote ourselves:

• To handle access controls as well as the developer-facing repo management knobs (personal server-side clones, personal scratch repos, access management for those, a trashcan for repos, the works) we chose gitolite, which was definitely the best choice we could have made. It’s reliable, well-documented, fast and rich in features, and its principal developer Sitaram Chamarty has been an incredible aid to our efforts both with many hours of work spent coding additional features we needed and tons of advice. We did write a couple of add-ins to gitolite ourselves to provide some additional commands to developers, which we found pleasant to do.
• As web-based repository viewer we use an inhouse-modified version of Redmine. As with gitolite, we’ve found the Redmine crew to be easy to approach and get help from, and while we ran into some trouble with the scalability of its Git support, it proved easy enough to modify without requiring massive forking. Redmine’s database of projects serves a vital role in the system in that the gitolite config is generated from Redmine’s database (e.g user access roles), and we also generate a custom-format XML from it that is used by end-user clone/build tools, our translation infrastructure, LXR and many other bits and pieces that have a need to discover projects and repositories and their metadata (which can be modified by project managers in the Redmine web UI, e.g. setting description texts or translation branches).
• For review, we use ReviewBoard, continuing on from SVN. This is definitely the weak spot in our setup right now; ReviewBoard just doesn’t support Git very well since it’s designed to handle single patches rather than series of them and thus also can’t track branches. We’ve been looking at Gerrit on and off to replace it, but so far can’t really bring ourselves to commit to it since it’s a fairly unwieldy beast.
• We wrote our own repository hooks in Python, which are in charge of things like sending out commit mails to mailing lists, CIA.vc, the bug tracker and people mentioned in commit messages behind special keywords, evaluating other keywords to close bug tickets or review requests, check for trivial errors (EOL style, license headers, that sort of thing), ref backups and a bunch of other tasks. Some of which run through asynchronous job queues as required to avoid blocking.
• We wrote our own mirroring system to provide several read-only git:// mirror servers behind a geo-ip DNS load balancer. The mirrors get updated whenever a push to the master happens.
• KDE has an LDAP-based single sign-on system in use at many of its web properties including Redmine and ReviewBoard, currently using an inhouse-modified (mostly expanded with our own modules) version of GOsa as web frontend. GOsa is also used to request developer privileges and manage the SSH keys for Git write access.
• There are a couple of other minor bits and pieces flying about, like a commits.kde.org/ web app written in Ruby+Sinatra that forwards to the Redmine page for the commit; those URLs pop up in various places, e.g. in a developer’s terminal output after pushing, in mail headers, CIA.vc postings, etc.

KDE’s migration to Git is still ongoing, with many large and actively developed codebases still remaining in SVN at this time, as well as the entire translation effort. To actually migrate things we use our own tool called svn2git which takes raw SVN repo data, an account conversion map (supplied as plain text generated from our LDAP) and a plain text file with regex-based rules as input and produces git repositories. Writing the necessary rules files is unfortunately a time-intensive task especially given some very complex project histories originally imported from CVS, but additional tools exist to partially auto-generate them by analyzing SVN history programmatically.

Here’s a couple of links of interest:

• The developer-facing user manual for KDE’s Git infra: http://community.kde.org/Sysadmin/GitKdeOrgManual
• The initial posting of the sysadmin working group’s plan for the above setup, and our evaluation of this stack vs. the Gitorious software (ODF of the latter is attached to the first reply): http://mail.kde.org/pipermail/kde-scm-interest/2010-June/001…
• The XML export from Redmine mentioned above: https://projects.kde.org/kde_projects.xml

But the thing is, this feature was discussed a long time ago, see for example the famous sysadmin advice. The xml-file making it possible for kdesrc-build to work like it does was developed shortly after the first extragear applications started the migration to git. With great input from Eike Hein and implemented by Ben Cooksley. I want to give them both a big Thank You as well! Well done.