We have three mirrors for both. One not located in europe and two are. If you are europe you want to use the ones in europe and if you are physically closer to the third mirror, you want to use that. That improves everyones latency and probably makes anongit/anonsvn a bit faster, as you should always get the server which is close to you.

We have not heard anyone complaining about this change, so I figure that the best compliment you can get. It also means that we are now able to take out mirrors and add new mirrors within some minutes. In the previous setup it took us 2-3 hours for the nameservers to spread the news. Which sucks if a mirror suddenly has dropped out. This is now solved.

Another option we have is to push load info to the system, the system can then adapt and traffic some more visitors to a server with lesser load. We have not yet implemented that, but it’s nice that we have this option.

We will also explore the possibility to use this system for the ftp-mirrors, so ftp.kde.org will return automatically a mirror close to you. But if that’s going to happen remains to be seen.

Anyhow, you can do a quick test how far away you are from a anongit/anonsvn mirror by executing dig -t txt anonsvn.local-kde.org @ns1.geoscaling.com, this will return the city you are in and the mirror you will be using. Please note that in many cases the city is a bit off, but that does not really matter as we don’t have so much mirrors that it would really make a difference. Enjoy.

This service was kindly donated by GeoScaling. They are providing such a service for free currently. The support was awesome and fast. They even started to chat with me on the site while I was setting up the system, asking if I had any problems. I wish all companies would be like this.

We have several hooks. There is one that checks the end of line signs, one that checks for invalid licenses, but also the ones that deal with keywords, like the one for closing bugs, or CC’ing someone. That brings us to another set of hooks, the ones that sent out mails to CIA.vc and the commit mailinglist.

In the past months we have worked with the hooks written in the past for gitorious, if I’m correct. In the meanwhile we have detected multiple problems, which began to build up in the last few months. For example keeping the whole diff in memory for all the processing is convenient, and also very possible in projects like Amarok and Konversation, but now kdepim is also using it, we reached the memory limit quite often, which broke stuff. We have solved that now, and used a quite intensive set of kdepim commits from the past to make sure the new hooks were up to the task.

We also tried to match the existing mail layout as we use it for svn. The git repo mails where a bit messy, and that’s been cleaned up now. We have added new headers, like X-Commit-Ref, X-Commit-Project and X-Commit-Folders, so that makes it possible for commitfilter to be able to support kde repos natively. Especially the lack of X-Commit-Folders broke many people’s filters.

The keywords syntax has been improved, some of the syntax that was supported for svn commits did not work for git, we hope to have corrected them, so they work now. And we moved from bash+ruby+perl tangled mess to pure python, which makes it way better maintainable for everyone.

I’m writing ‘we’, but as usual, I did not do much. All credits go to Ben Cooksley, so all comments should go to his first blog. He is one of the people that works pretty silently, but does amazing work. If I could propose someone for an Akademy Award, I would definitively put him on the short list. Luca Beltrame (or we rather know him as einar77) also helped, so give him a beer if you see him and sent the bill to me. Hmm. I might take that back.

Anyhow, enjoy the new hooks, and if you spot regressions, feel free to contact us.

For the last couple years, I ran KDE from trunk from svn. That meant that I only had a need for a basic Linux OS. Debian was fine for that. Combined with Equives I could still use some distro packages while compiling most of it myself. But there were problems. Mostly these were caused by my own lazyness in configuring the right stuff in the right way.

Dealing with Virtuoso, Mysql embedded, OSS, Phonon, packagekit, consolekit, hal, udisk, polkit and all that stuff became a boring and silly exercise, I never got it all right, which meant working with no sound for a while, having no permission to change the time from within KDE and that kind of little annoying setup bugs. Compiling the stuff became harder too, as you sometimes only noticed when compiling kdebase that a kdesupport lib had moved to git and you had to fix that.

But my life has changed too. I no longer hack on various stuff, I’m happy to work as a KDE sysadmin with a great group of people. The only thing I want to keep working on now and then is RSIBreak, but for that I don’t exactly need the whole of KDE from SVN. But I also do some work for the Release Team, so I can not be running to behind with the KDE version I’m running.

That made me decide to switch away from Debian. They are doing great work on KDE and I’m sad to move away, but they just don’t provide bleeding edge packages currently. Which is absolutely understandable as they have focus on a rock solid stable experience for users. I choose Fedora because I was told they rock in packaging KDE and are providing packages days after releases.

And I’m so happy so far. Installing it was kind of hard as I ended up in GNome, without having indication how to switch to KDM and KDE. But after solving that, I’m pretty happy. There is stuff to explore, like those SELinux warnings (yes, this time I did not disable it right after installing the distro) and there is stuff that just works. Like sound. I’ve no idea if I’m running OSS or PulseAudio, or which phonon backend I’m using. And I don’t care, it just works. Just like polkit, which I never ever got to work.

So up to now, big compliments to the Fedora team. Nice job, and that needs to be said too sometimes!

This blog is a call to find someone who wants to implement this for us. The basics are there, there is a REST API for reviewboard which should allow this.

We also have hooks in place for git and svn so those can be extended. Some things that needs to be resolved by the one doing the work:

• write the script to fire the api call to reviewboard
• find a way around the fact that it looks like you have to be logged in before you can close the request.
• make sure the script is non-blocking, that means the one committing or pushing should not have to wait untill the results of the api call have completed, should happen in the background
• there is a reviewboard for git and svn, both need to work, so the hook for each need to be adjusted.

If you are interested in implementing it, come to the #kde-sysadmin channel, and I’ll add you to the relevant bug report!

I can see from closeby how much work is involved with making all this happen in the first place, all the coordination involved, making sure no translations are lost, former svn locations cleaned up, review requests can be filed and kde-commits mailinglists does not get flooded on the initial push, with or without tags. I take my hat of for all the people that are doing all this work silently, and professionally.

For now only reasonably small projects have converted, but it won’t be long before this will change. I slowely see requests for whole modules come in. kdeplasma-addons and kdepim, all have concrete plans to move to git. I also saw some test conversions for kdelibs, so that will happen too. So, be prepared :). I guess it won’t be long before most KDE modules will switch.

We’ve been kept a bit to busy with all these conversions, we’ve failed to generate xml-files up to now for example. That would make it easier for tools to fetch repo’s, or hook in lxr and scripty into that. For example that would enable developers to turn of scripty for a while for their project by using a switch in the projects.kde.org site. Also applying consistent theming to all the sites has failed. We seriously need more people in #kde-www. I’m sure we will get to all this some day…

For example, with the amount of projects increasing, you can also see the fun bits, like the news page http://projects.kde.org/news, which is an aggregation of news items of all the projects. Think it’s about time this feed gets added to dot.kde.org or finds a place on www.kde.org.

Also improved this week is the project overview page, see http://projects.kde.org/projects. Not only has the hierarchy in modules returned, but the page itself has been pimped beyond recognition. The hierarchy required Ben Cooksley to hack on Redmine, we will feed those modifications back upstream. Impressive work there.

Now the hierarchy has returned, some interesting pages have appeared, like http://projects.kde.org/projects/extragear/activity. On this page you can see all activity which happened in extragear today. You can also add ‘/multimedia’ before ‘/activity’ to only see the activity in the multimedia section of extragear. This gives interesting opportunities for, for example, extragear.kde.org to integrate the RSS feed, to make it somewhat more dynamic.

This also is the start of the next step, which is to write software which will automatically create a XML-file describing all the projects. This XML-file will be used by Scripty, LXR, EBN to know the location of the different projects. Also kdesrc-build can use this file. So, if you want to know how that will look like, talk to us (well, Eike Hein (Sho_), he is on a mission) on #kde-sysadmin on Freenode.

For example in the future, it will be possible to turn Scripty on and off via a setting in projects.kde.org. Or indicate which branch is the stable branch, so translators know which branch has to be translated as well. Something that is kind of messy right now for extragear applications.

Another page that is interesting is when you click on your own name on the top right navigation. It will show you in which projects you are involved, and shows your recent activity (OMG, was my last commit in kdesupport/akonadi really on 2008-06-18???). Again, everyone with an account on http://identity.kde.org has access with the those credentials.

I went to the Techbase article at http://techbase.kde.org/Projects/MoveToGit/UsingSvn2Git and started reading. In the middle it just mentions to download 65GB. Hmm. Wait 65GB??? Even for me that’s a lot. The rest of the evening was spent watching rsync pulling in the data.

I felt that this situation was unacceptable, so last week I blogged that we would like to setup a couple servers for people to work on, with that 65GB already checked out. The responses where very good. Several individuals, companies and distro’s approached us and did some amazing donations. We’ve accepted three servers and prepared them.

The three servers are now available for all registered KDE developers. With the ssh key you have uploaded to identity.kde.org, you also have access to Huey, Dewey and Louie. They are identical in setup. I think this is the first time we provide ssh access to machines, so this is also for us an exciting step. Hope we did it right.

Read the rules and instructions at http://community.kde.org/Sysadmin/DeveloperAccessForRuleWriting. Have fun writing the rules and converting your subversion application to git!

To convert a project from SVN to Git, you have to write conversion rules. To write these rules and test the conversion you need to have the complete SVN repository on your local computer. Even if you only would like a small contained project like RSIBreak to switch to git, you need to rsync the whole repository.

The size of that repository is currently 65GB, this is a real problem for a lot of our contributors. They can not download this repository with their ADSL connection. If you have a 2mbps internet connection which you can use 100% and calculate 0% overhead, this will take you 70 hours, 31 minutes and 46 seconds. That’s basically unacceptable and blocking the transition to Git.

So, our plan is to provide a server on which all our registered developers can login, which is completely setup to write the rules and has the SVN repository downloaded and kept up to date. This way all the developers can log in to that server and start working on the rules instantly.

But currently we don’t have a server which we can use for such an isolated purpose. Ideally we would like a (perhaps virtual) machine with something like 100GB free space with some fast access times to the disk, as that 65GB will be searched through frequently. Also a good processor makes things more enjoyable. We are hoping data centers, commercial virtual server providers or some friendly distributions are prepared to donate some resources to us.

If you have hardware available for us, please drop by #kde-sysadmin on freenode, or sent me a mail (toma at kde ). Ideally we would like to have two or three servers available for this project and we only need it for a year or so. Please help us, help KDE!

The site is powered by Redmine software. It’s very powerful software which can also be used for bug reporting, bug tracking, time tracking, file management, wiki and lots more. But for now there is no plan to use all these features.

An important link is a link I rather not give, but have to do, it is the link to the overview of all projects. I rather not give it, because the current layout is less than crap, I hope we will be able to theme it soonish. It can be a lot better, but that’s work in progress.

After you click through one of the projects, you can basically see 3 or 4 tabs. One of them is the activity-tab. It contains a summary of activity within the projects, basically an aggregation of the commits and news for now, an RSS feed is provided, so that’s the RSS feed you want to be kept up to date about everything that is happening.

News
Each project can use the News section to publish the latest news for that project. You can announce new releases or use it as a project blog, or whatever you like. The fun part is that the homepage of projects.kde.org will show the latest news, so you can promote your project. Of course a RSS feed for the news is available. Currently there is no project which has provided news, but I invite everyone to use it, so the homepage gets a bit more interesting.

Repository viewer
Another tab leads to the repository viewer. Let’s take a look at the repository of Yakuake (I know Eike likes it to be used as an example :)), see http://projects.kde.org/projects/yakuake/repository. The first important block is that at top, it explains the way you can clone the git repository. It also indicates which methods give you read-only access and which gives you read-write access. Read-write access is available for all kde-developers. I’ve said it before, but if you have a svn account, you also have write access to all KDE git repositories.

Below that block, there is the ‘filesystem’ display which you can browse. Pretty similar to what websvn.kde.org provides for our subversion repository. It has pretty much all the features you would need to browse the source, switch to a branch or to find out who is to blame for a certain line of code. And all that a lot faster than websvn.kde.org.

Below that there is a list of the last commits that have taken place. You can just hit ‘View differences’ to see the difference for the last commit. You can see the diff inline, but also next to each other or as unified diff. There is an RSS feed as well, so you can keep an eye on all the changes easily, without needing commitfilter for example.

Also nice are the statistics. It gives you a quick overview about how active the project has been in the last couple of months, or to get an idea about who is involved in the project. Unfortunately for bigger repositories like Amarok this feature is pretty much unusable, due to the high amount of different committers, but I’m sure we get around talking to upstream about that one day.

That leaves the homepage of each project. That’s the main place for your project, it should contain a description and links to common development places for your project, like the bug tracker, irc channel or whatever.

When you request a git repository, you get the project page automatically. You don’t have to use the project page, but you are able to use it if you like. Certainly for small projects it could replace the usual one page website that gets created for it. For example RSIBreak, it would not need to use a website like rsibreak.org (btw, released a new beta last week, try it!), but simply use the project page to provide all the info for fellow developers. Especially if we enable more features in the future, for example if we connect the file management Redmine module with ftp.kde.org, just to name one of my secret plans…

With opening the site for everyone, the way new developers get commit access has changed. From now on you register on identity.kde.org, and after you login, you will find a form to apply for ‘developer status’, which will ask you for your ssh key, asks a justification and asks for a supporter.

After the supporter has given his support to the request, we move the user from the ‘users’ group to the ‘developers’ group and that will give the person commit access to SVN and push access to GIT.

If you look at the Schedule this are the items originally planned for the 15th and the 22nd, but the administrative overhead of the current situation was high and the scripts were ready, so we launched it early. I just created the first account via this new procedure and that went smooth.

Next item on our todo list is to launch the reviewboard for git repositories and redmine. Both will launch next wednesday and you can get access to both systems with the same login and password you use for identity.kde.org