Move to Git: The Progress so far…
This blog is written by a couple of sysadmins together, where we want to give a small update on the move of KDE to git.
The document we posted a few weeks ago on the scm-interest mailinglist presented a couple of software solutions we want to use for the migration of KDE git. Let’s see what happened since then:
Gitolite
Gitolite is the heart of our migration. Compare it to our svn server. It hosts our repository. We have setup gitolite and made conversion scripts to import all the keys of the existing developers to have access.
We realized we had a problem however with users which do not use ssh keys but use a password instead. To fix this, we have written a webpage allowing developers to migrate smoothly from passwords to ssh keys. We also invited each and every developer to migrate and had a complete hectic weekend in which we migrated a large amount of active developers.
We also configured gitolite in the way we wanted it to be setup. The author of gitolite is in our #kde-sysadmin channel, and jumps up and down when we tend to do something stupid.
GOsa
We realized that our document was not quite complete. The problem was that we acknowledged that there needs to be a single point of authentication. The separate parts of our stack need to be presented to the developers as one. That means that the different services would not require separate login and password registrations. That’s where GOsa enters. It provides a web interface around a LDAP directory. There we can store a password once and use that on Reviewboard and Redmine at first.
We plan to extend this to other websites outside this over time, with all websites eventually using this central source for authentication. This means that only one registration will be needed to access all KDE websites. We also plan to allow developers to manage their SSH keys through this service.
GOsa will become available at auth.kde.org ( btw, GOsa can be found at http://gosa.gonicus.de/ ) which we have modified to allow anonymous registration, and the ability to reset your password without needing a sysadmin. GOsa is responsible for controlling what data can be changed in the LDAP directory, and by whom. Upgrading from a User to a Developer with push access to gitolite, will be as simple as adding the user to the Developers group.
In the past few days we have setup, modified and extended GOsa and have adjusted Redmine to use the LDAP directory managed by GOsa succesfully.
OpenID
Unfortunately we had to come to the solution that relying on OpenID for all our services could not be implemented currently, we have been keeping it in mind while setting things up, so we will revisit this topic soon after the initial implementation we are doing right now.
Redmine
Redmine has been setup at http://projects.kde.org. This went pretty much as expected and works fine. We just needed to disable the possibility for public registration and for existing developers to change their name, email and other details, which has now been completed, as those details are all managed by GOsa.
Reviewboard
We ran a somewhat older version of Reviewboard. We contacted David Solbach and he quickly came to our IRC channel, got highly motivated and started working on it. One of the problems we faced is that the current Reviewboard contained over 3500 accounts made by spammers. We cleaned that up. When the final move is near, we probably need to setup a new Reviewboard instance, as the existing usernames are likely to clash with the usernames coming from the LDAP directory.
cgit
Problem. We discovered that cgit is not up to the task we wanted it to be, although it looks clean and is super fast. Several patches were not displayed properly. We are now experimenting with gitweb to see if that works better. It’s not a big deal, as this part is not crucial for our setup, it just provides an additional view of the repository, next to Redmine.
When can we migrate?
The key question of many people is when we are ready. Currently we still have some things that need to be finished before we can begin migrating, among other things these are:
- hook Reviewboard into our LDAP setup
- testing the complete setup, including failure scenario’s of the LDAP setup
- database corruption
- ldap connection problems
- slave syncing problems
- moving the services to their final destinations
- talk to the maintainers of our anonsvn setup, to create anongit.
- finished git conversion rules
- well, we are not going to do this ourselfes, but it needs to be done before we can migrate the whole of KDE.
- adapt some minimal KDE stylesheets and logo’s to the new services
- gazillion smaller stuff
Conclusion
Overall, we consider that significant progress has been made towards commencing the move from SVN to Git, with further progress is still to be made however before we can move.
Hey!
Great stuff you’re using GOsa… have been doing my internship over at Gonicus and had a great time testing that :D
Lukas
Very nice, looking forward to being able to use git :)
Re. cgit: Can you describe the problems in more detail? What kind of patches were not displayed properly, and what’s the underlying issue?
I’m a heavy cgit user, and contribute patches to it once in a while, so I may be able to look into this.
Thanks for the report. It’s good to see that you are making so much progress. Great work.
Thanks for the great work. I would rather try to get the cgit bugs fixed and missing features implemented than going with gitweb. gitweb is a big and ugly monster :)
How will public-key only developers create their initial account on GOsa? E-mail verification?
Yes, we will create all the accounts automatically, based on the kde-common/accounts file. Developers can then use the lost password functionality to gain initial access. Maybe not super pretty, but it will work.
omgwtfbbq! single signon for *base? :) :)
*HUGS*
@Chani No, not Single Sing On, but Single Sign In. That means you can use the same credentials among the different services, but it does not mean, you are automatically logged in on all the services directly by entering the credentials once.
Note that our main repository browser is always going to be Redmine; be it cgit or gitweb, they’re both just “bonus”.
Johan:
1. It’s not displaying this blob correctly: http://cgit.kde.org/amarok/amarok/tree/README (“plain” works)
2. It’s showing a bogus diff for this merge commit (while git show, Redmine, Gitorious and gitweb show none, as expected): http://cgit.kde.org/konversation/konversation/commit/?id=7d1cb2fae86290da76ee0d0ded602c777f6e0ac8
3. The caching appears to be wonky in practice, e.g. with caching enabled it won’t update the Branch table on the “summary” page to show the actual latest commit.
It is about time.
Even gnome managed to migrate o_O.
Keep up the good work.
@Eike: Thanks for details. The links you gave return 404, but I tweaked the URL to git://git.kde.org/… and was able to clone the repos onto the machine where I play around with cgit.
Re. issue #1: Unable to reproduce. AFAICS on my cgit install (unfortunately not accessible from outside my company’s firewall), the blob displays correctly. I copy-pasted the text to a file, and diffed it against the ‘plain’ download, and there was no diff (except for a couple of lines with trailing whitespace in the plain version (probably “removed” by the browser in the ‘tree’ view))
Re. issue #2: Reproduced. AFAICS, cgit always shows the diff against the first parent, both for merge commit and regular commits (where the first parent is the only parent). However, ‘git show’ (and also Redmine, Gitorious and gitweb, I assume) uses the “combined” diff format when displaying merge commits (which often shows an empty diff for merges without conflicts). It should be fairly straightforward to add the same behaviour to cgit. I’ll try to look more into it, hopefully next week.
Re. issue #3: AFAICS the caching in cgit maps a request (URL with query parameters) to a cached response. If the underlying repo has changed in the meantime, it seems the cached response is not invalidated. Rather it will return the cached/stale response until it expires (by default after 15 minutes IINM, although this can be configured). I’m not sure whether this should be considered a bug in the caching code, or if it’s seen as a necessary tradeoff when enabling caching. In any case, I suspect that even is you disable caching, cgit will still beat gitweb performance-wise with a comfortable margin.
@Johan: Thanks for taking a look, some more from our end:
Issue #1: The issue was that the HTML view basically showed the blob as all newlines – it displayed as many line numbers as the file had, but all of the lines empty. Not a browser-side display issue, either; the source served up outright didn’t contain the content.
Issue #2: The problem with the current behavior is that the diff displays an author together with changes the author didn’t make, so we ended up with people who got upset that cgit seemed to attribute actions to them that they didn’t perform. It’s a social issue as much as anything; the git show behavior communicates better that it’s just a merge.
Issue #3: What we saw was that it didn’t update the mentioned place in ~20 hours or so, but maybe we had the expiry knob turned the wrong way (though afaik we were using default settings).