Toma's blog » sysadmin

[sysadmin] Status Update

toma — Sat, 24 Jul 2010 23:28:17 +0000

We thought it might be a good idea to give everyone a new status update about the progress we are making regarding the conversion to Git. This report has been put together by a couple of members of the sysadmin team. Let’s start with the tasks done in the past weeks:

SSH Key Management itself can be done in Gosa – but the backend to get it live with gitolite is still to be done (see current todo list below)
Integration of Reviewboard with LDAP has been accomplished in a testing instance successfully. A script has also been written to sync data such as names and email addresses from LDAP into both Redmine and Reviewboard.
Gosa has begun to be themed using the kde.org style, chihuahua. Ingo Malchow is brave enough to work on this part.
Ben Cooksley has been working on making some additional forms for the initial account registration. Basically the new workflow will be like this:
1. User can register, and can use their account immediately after verifying their email address.
2. User immediately gets access to redmine and reviewboard and gosa with one and the same login credentials. This is called Single Sign In.
3. User can change his own data in gosa, for example his ssh public keys. Add more keys or replace lost ones. (yay for no more sysadmin involvement for that)
4. If the user wants push access to the git repositories (or SVN while it’s still around), he has to apply for that. This is similar to the current form; indicate a supporter and write a small justification.
5. After evaluation and approval from the supporter, we add the user to the Developers group in gosa, and the user immediately gets push access.
  This grouping mechanism can be extended in the future, to have – for example – a good address book like system for eV-members, or to upgrade users to sysadmins.

The current todo list is basically:

drive redmine project and reviewboard group creation from gitolite
This will be used to allow any project to instantly begin to recieve reviews, without any further effort being needed.
redmine and rb project namespace problems
At the moment, both Redmine and Reviewboard have problems with projects using the same name, which when you have sub-projects to create a tree of projects, becomes problematic.
A script to create GOsa accounts, insert SSH key(s) + mail address from the existing data
Modification of Gitolite to read from LDAP for SSH keys
By default Gitolite reads the allowed keys from the git admin repository, but we have decided to put the keys in an LDAP database, so there have to be a process to copy the keys to the right place. Currently we do it manually, but that’s not a solution we like :) We just discussed this with Sitaram Chamarty, the author of gitolite and came up with the following workflow:
- When a user changes a key in gosa, automatically a script is called on the gitolite server so we know something has changed for that user.
- A script extracts the keys from the LDAP database and updates a folder where all the keys are.
- Once this is done the internal list of authorised keys will be updated, allowing the new SSH key to be used immediately by the developer.
  That means the keys wont need to go into the special git admin repository which is only accessible by sysadmins. Disadvantage is that this change has to be written by Sitaram with our assistance. So give Sitaram cookies when you see him.
- Use gitolite for SVN authentication, tying SVN into the unified account system
- Retire the old “Get an SVN account” web app (as Gosa takes its place)
- Send out a final set of ‘convert to ssh keys or else your svn access will stop working’ invitations for svn users which use a password currently.
- Move all the software bits to the right hardware, and give everything the final shake down.
This blog is a bit on the technical side probably. The open todo’s unfortunately are rather technical, so that’s unavoidable. But it still gives you a good idea about the progress and the things we are currently facing.

Move to Git: The Progress so far…

toma — Tue, 06 Jul 2010 22:08:16 +0000

This blog is written by a couple of sysadmins together, where we want to give a small update on the move of KDE to git.

The document we posted a few weeks ago on the scm-interest mailinglist presented a couple of software solutions we want to use for the migration of KDE git. Let’s see what happened since then:

Gitolite
Gitolite is the heart of our migration. Compare it to our svn server. It hosts our repository. We have setup gitolite and made conversion scripts to import all the keys of the existing developers to have access.

We realized we had a problem however with users which do not use ssh keys but use a password instead. To fix this, we have written a webpage allowing developers to migrate smoothly from passwords to ssh keys. We also invited each and every developer to migrate and had a complete hectic weekend in which we migrated a large amount of active developers.

We also configured gitolite in the way we wanted it to be setup. The author of gitolite is in our #kde-sysadmin channel, and jumps up and down when we tend to do something stupid.

GOsa
We realized that our document was not quite complete. The problem was that we acknowledged that there needs to be a single point of authentication. The separate parts of our stack need to be presented to the developers as one. That means that the different services would not require separate login and password registrations. That’s where GOsa enters. It provides a web interface around a LDAP directory. There we can store a password once and use that on Reviewboard and Redmine at first.

We plan to extend this to other websites outside this over time, with all websites eventually using this central source for authentication. This means that only one registration will be needed to access all KDE websites. We also plan to allow developers to manage their SSH keys through this service.

GOsa will become available at auth.kde.org ( btw, GOsa can be found at http://gosa.gonicus.de/ ) which we have modified to allow anonymous registration, and the ability to reset your password without needing a sysadmin. GOsa is responsible for controlling what data can be changed in the LDAP directory, and by whom. Upgrading from a User to a Developer with push access to gitolite, will be as simple as adding the user to the Developers group.

In the past few days we have setup, modified and extended GOsa and have adjusted Redmine to use the LDAP directory managed by GOsa succesfully.

OpenID
Unfortunately we had to come to the solution that relying on OpenID for all our services could not be implemented currently, we have been keeping it in mind while setting things up, so we will revisit this topic soon after the initial implementation we are doing right now.

Redmine
Redmine has been setup at http://projects.kde.org. This went pretty much as expected and works fine. We just needed to disable the possibility for public registration and for existing developers to change their name, email and other details, which has now been completed, as those details are all managed by GOsa.

Reviewboard
We ran a somewhat older version of Reviewboard. We contacted David Solbach and he quickly came to our IRC channel, got highly motivated and started working on it. One of the problems we faced is that the current Reviewboard contained over 3500 accounts made by spammers. We cleaned that up. When the final move is near, we probably need to setup a new Reviewboard instance, as the existing usernames are likely to clash with the usernames coming from the LDAP directory.

cgit
Problem. We discovered that cgit is not up to the task we wanted it to be, although it looks clean and is super fast. Several patches were not displayed properly. We are now experimenting with gitweb to see if that works better. It’s not a big deal, as this part is not crucial for our setup, it just provides an additional view of the repository, next to Redmine.

When can we migrate?
The key question of many people is when we are ready. Currently we still have some things that need to be finished before we can begin migrating, among other things these are:

hook Reviewboard into our LDAP setup
testing the complete setup, including failure scenario’s of the LDAP setup

database corruption
ldap connection problems
slave syncing problems

moving the services to their final destinations
talk to the maintainers of our anonsvn setup, to create anongit.
finished git conversion rules

well, we are not going to do this ourselfes, but it needs to be done before we can migrate the whole of KDE.

adapt some minimal KDE stylesheets and logo’s to the new services
gazillion smaller stuff

Conclusion
Overall, we consider that significant progress has been made towards commencing the move from SVN to Git, with further progress is still to be made however before we can move.

Howto (not) communicate with sysadmin

toma — Fri, 02 Jul 2010 22:21:25 +0000

Recently I’ve seen all kinds of communication with the sysadmin team that simply does not work. Here are some examples:

Using random irc channels like #kde-devel for asking if systems are down. If you want to know that, use #kde-sysadmin.
Assuming sysadmin knows of a problem. Sometime I only find out that a server is down after a few hours. Simply because everyone assumes someone else has reported it. Don’t ever assume it. Report a bug at https://bugs.kde.org/enter_sysadmin_request.cgi (it is in the footer of bugs.kde.org, so very simple to find
A variation of that is silently assuming things. Say you tell us Foo and that we should deduct that Bar needs to be done and we undersand that. No we don’t. Be explicit, file a sysadmin bugreport.
Trying to speed up things by filing a bug and approach us privately. No it won’t help, we usually process bugs very very quickly. Last week this caused double work, as Dirk was on irc and I was not and only saw the bugreport. So we did it twice. Really a waste of time.
Sending a mail to sysadmin at kde dot org. This has a high chance to get lost and not processed at all. If you are lucky you get a mail back to report a bug in bugzilla. So why not use it in the first place?

I know communication is tough, but to keep it structured and managable, just file sysadmin bugreports. It is very simple (no wizard with multiple steps and duplication checks, just a one page web form) and it would help us a lot!

SVN Account Conversion FAQ

toma — Sun, 27 Jun 2010 16:57:45 +0000

Not that there are many questions about the conversion of password based svn accounts to ssh-based ones, but the FAQ-format is nice to address a bunch of random questions.

After converting my account, it asks for a password when committing, and I don’t know what to give in. Help?
The point here is that the svn switch command we have given you has not been completed successfully. Unfortunately svn does not tell you that. What you need to do is run svn info and see what the URL is. In most cases the problem is that the username is not in the URL. So you need to use the following command: svn switch –relocate https://svn.kde.org svn+ssh://username@svn.kde.org

The first argument after –relocate must match the URL found in the svn info command.

You have failed to setup my account properly, it closes the connection as soon as I try to commit anything.
Yeah, another nice uninformative error message. The problem is that you have an offending key for svn.kde.org in you known hosts file. Try doing ’ssh username@svn.kde.org’. If that result in something that looks like garbage, it is ok. Else it will give an error message which shows which line in your .ssh/known_hosts is offending. Remove that line and try again.

Note, only remove that line if you run into this problem right after conversion, if you are already using ssh based svn and get this error suddenly, chances are high there is a man in the middle attack.

Is rsa better than dsa?
Google for that one :)

I use git-svn, can I convert my existing checkout?
Hmwah, yes, you can, but it is not straight forward. Some followed this tutorial with success. You should use svn+ssh://username@svn.kde.org/home/kde format as the new url. We have a sysadmin bugreport with some more detailed information. We can subscribe you when you want, just let us know on #kde-sysadmin.

Can I use multiple ssh keys for my svn account?
Yes You Can. Just file a sysadmin bugreport at this location. Make sure your bugzilla login matches your email address in the kde-common/accounts though.

Can I use the same ssh key on multiple computers?
Yes You Can. Please note that when copying your key files, you must make sure the permissions are exactly identical to the place you copied it from. Making your private key world readable and writable makes ssh refuse to use that key.

Converting, disabling & converting subversion accounts…

toma — Sat, 26 Jun 2010 10:17:43 +0000

The past 24 hours have been amazing. The sysadmin team has been busy continuously. Eike Hein was on duty yesterday most of the day, and when he (finally) decided to go to bed, Ben Cooksley took over. I guess night shifts are not so bad when you are living on the other end of the world. I’ll take the next shift if Ben has had enough.

We are busy converting password based subversion accounts to ssh based ones. The benefit of doing that is that you automatically get access to the KDE Git repository. As KDE is moving to Git, we need to either convert the password based accounts or disable them.

In the past 24 hours we have sent out mails to 1210 subversion account holders which use a password. Of those roughly 150 have replied to the invitation. Roughly 100 replied with their ssh key, 1 with his gpg key and about 50 replied that they want to close their account.

That might seem like a high number of people closing their account, but seeing it into perspective of the total accounts we have, and considering nobody gives up his or her account usually, I think it is actually a very nice number.

The 100 we converted were all converted instantly, nobody had to wait, which is a great achievement. Ah, Ben just asked me to take over, so this blog needs to be ended now. The work is comparable with creating new accounts, so we still need to actually do something for each developer that wants to convert.

What I wanted to say is, that if you have not received an invitation to convert, please go to this page and re-request your invitation. If you still don’t receive it, check if your email address is correct in the accounts file. If not, adapt it there, and re-request the invitation in a day or so.

If you have any questions, jump into #kde-sysadmin irc channel or file a sysadmin bugreport here.

Oh, backlog building up…. Bye!

Disable your account if you are no longer active

toma — Fri, 25 Jun 2010 17:47:26 +0000

My last blog triggered a handful of people who requested the merge from a password based svn account to a ssh based one. This was very nice, because we could iron out the last glitches in the scripts processing the requests.

We are now approaching the password based accounts directly. Everyone receives an invitation to convert. We send them out in managable batches, and we expect the last batch to go out later this weekend. We still have roughly 1200 accounts which need conversion.

Approaching everyone individually also means for some account holders a flash back to the times they were active for KDE. They could have been inactive for years already, as we never ‘clean’ svn accounts automatically based on inactivity.

If you are inactive you can also indicate that on the form, your account will be disabled and we will no longer spam you in the future to convert your account. So, if you want to pro-actively help us, feel free to go to this page and fill in your svn account name. You will receive instructions in your mailbox within seconds.

Convert password based accounts

toma — Wed, 23 Jun 2010 17:11:39 +0000

Traditionally we have two different account types within KDE. Some people use https://$username@svn.kde.org and some people use svn+ssh://$username@svn.kde.org. While we are moving to git we can not use the https version anymore, write access to the git repository will be based on ssh keys.

To streamline your move to git, you will have to convert your svn account from https (if you are using that) to svn+ssh by providing us with your ssh public key. If you do that, you will automatically get write access to the git repositories. If you already have a svn+ssh account, there is nothing you need to do right now.

To facilitate the move from password to ssh based accounts, we have created a special page where you can request the conversion.

Please note that if you are behind a firewall, which blocks port 22, you should not convert your account at this moment. We are working on a solution!

Important: Update your email address

toma — Sat, 12 Jun 2010 23:24:45 +0000

As you all might know or not, KDE is moving to git. With applying for an svn you had the option to choose for a https account with a password. With the move to git now in high gear we have to convert all those accounts to ssh accounts.

What will happen is that we will send every https user personal invitation to go a to a webpage. On this webpage you can upload your public ssh key or indicate that the account should be disabled, because you don’t use it anymore.

The invitations will be send in like two weeks. Every invitation that bounces will probably mean that the account will be disabled. So we can not stress enough that you check your email address which is associated with your account. You can find it in kde-common/accounts. You can simply do a checkout and commit changes to your own entry.

Ssh users dont have to do anything, your public keys will migrate automatically. Https users, please wait for the invitation, as we will automate most of the process to move you from https to ssh. So there is no need to request the move from https to ssh at KDE’s sysadmin, unless you are developer of the first couple test projects.

Advertisement:
This post is made on my new N900, with the MaStory app, wich is a totally cool app on a totally cool device. I just had to tell you :-)

Turn off those password reminders.

toma — Sat, 23 Jan 2010 21:20:09 +0000

If you are a mailman admin of any KDE mailing list, please consider turning off the monthly password reminders. Sending passwords in plain text is a bad idea to start with, doing it regularly without reason is even worse…

Login into the admin, go to the ‘General Options’, half way down there is a section ‘Notifications’, where the option is for the monthly password reminders.

It’s the default for new lists KDE sysadmin creates, but there are still a few mailing lists which have that bit set. If you need help, ping me on irc or file a sysadmin bugreport, the address is in the footer of bugs.kde.org.