Not only is this very difficult to maintain organisationally, hard to debug sometimes, causes confusion not only with sysadmin but also with the contributors, but is also a waste of resources (instead of 4 anongit mirrors, we can only have 2, because we also need 2 anonsvn mirrors).

KDE has decided to move to Git a few years ago. Now the time has come to say goodbye to subversion. That’s why we created a time schedule to dismantle subversion.

In short, I don’t think bugs.kde.org is going to be decommissioned ever, after I realized that, I also decided to step up and see if we can make the user experience better. And luckily Ingo was already improving bugs.kde.org step by step, Let me explain some of the changes, as they might be subtle and you might not have noticed them.

• ‘NEW” was renamed to ‘CONFIRMED’. New was confusing for users. A bug 10 years old, can hardly be seen as new. Confirmed describes it better, it is the next step beyond ‘UNCONFIRMED’.
• When a new bug gets entered, in the list of existing bugs, now a version number is visible. See for example here.
• We also added a dupe-count column, this way you can add this column to your queries and see how many people reported that bug or crash.
• The software is upgraded to the newest version.
• Tons of tiny changes, like that status of a bug, right below the main title. And some adjustments for people with small screens.
• In the mails bugs.kde.org send out, the topic now starts with the name of the ‘product’,
• If you add a comment to a bug, you wont go to the next one in your list, but you will go back to the same bug. This was confusing for a lot of new users. You can still return to the old behavior in the settings.

All these are triggered by the bugs that are reported against the bugs.kde.org-product. We are slowly going through all those reports, answering, closing, etc. If you have any wishes, just go there, see if it’s listed already and comment on them or file new ones. We want to make it the best bug tool we can possibly can.

You connect your IRC client to the bouncer and you use it as usual. As soon as you close your IRC client, it will store all the traffic to the channels. After you start your IRC client again, all that traffic will be replayed. This way you can easily catch up with everything that happened while you were disconnected.

Most bouncers are difficult to tune and to set up. We use the ZNC-bouncer and this comes with an easy to use web interface where you can configure some settings with a few clicks.

Anyhow, more information on our Sysadmin wiki, registered developers and well known contributors can request it as usual.

Every single second of every single minute, each hour and every day, holidays included, automated scripts are scanning the web. The scripts are looking for vulnerabilities. In the past, when they found one, they exploited that vulnerability almost immediately. They started to deface that website, use that server to scan for other websites with vulnerabilities or infect the site with malware.

But the Method of Operation has changed. Instead of immediately exploiting the website, they only plant a backdoor. A backdoor is a script or page that will give them access at any time. This has two advantages. First advantage is that by immediately exploiting the website by defacing it, it becomes immediately visible for the website owner. That means the hoster can exactly see how the infection happened and ban the IP-address of the attacker, or even report that IP, making the whole ‘bot’ unusable as they need an IP to continue the scans.

The second reason is that the one planting the backdoor can plant this on, for example, 100 sites. After he has done that, he could simply sell that 100 sites to a customer. Just like a list of credit card numbers, I guess a list of compromised sites is worth money too.

I’ve looked at some backdoors recently and what always strikes me is the level of maturity of those scripts. I’ve seen modules for popular CMS’s from commercial companies that contain far less quality code than most backdoors. They include user management, nice graphics, easy and user friendly interface and nice features. The backdoor is also hidden nicely. The header of the backdoor is nicely copied from surrounding files. For example a backdoor in Joomla contained all the official ‘Open Source Matters’ headers, including dates and revisions, as you would see in all Joomla files. It contains functions as usual, but somewhere in the middle of the file it contains an encoded block of code, which is the evil bit. You have to look twice to make sure you are not deleting an official file.

One of the advantages of splitting the infection and the exploitation is that detection does not happen immediately. This is nice because most providers do not store log files indefinitely. The planting of the back door could have happened in March, and the exploitation of that back door can happen in December. How many providers can find when and how the infection has happened? Not so many. And how about the fix? Should we go back with a backup to March? We can’t, so you need to fix it without exactly knowing how it happened.

The actual exploitation becomes smarter too. I’ve seen a nice one this week. The infection was done via a .htaccess file. They made nice mod_rewrite rules to only infect visitors coming from a search engine. Making the detection by the website owner less likely (as they would probably type in the address directly or use a bookmark).

This blog is mostly to make you aware that hackers are smart, your site could already be compromised without you knowing it. If you have not installed updates for your CMS, do it now, and don’t forget the plugins you installed.

Apart from the previous, you can also consider one or more of the following additional solutions to reduce the risks: add good detection software. You can look at JMonitoring for Joomla sites for example. It will notify you when one of the core files changes. Another solution is to try to stop the infection by using Incapsula. They scan for ‘bad bots’ and you can block them (we use this within KDE for some sites). Another solution is to put your CMS in Git or SVN and see what has changed each day. Be aware that files that appear in cache folders is also a popular place to put backdoors as people often discard them as automatically generated cache files.

Anyhow, take care, there is a lot of evil out there.

As we could recognize that bot because it went through all subscription in a speed no human does. We wrote some fail2ban script that would raise a firewall for that bot. That helped. But only for a day or so. Now we saw the same but every time the firewall was raised, it switched to a new ip-number, and the subscriptions continue.

The mailman software can’t cope with this currently. The only thing we can do is to completely block subscriptions via the web. We have now replaced it with a text for the users:

Web based subscriptions to all kde.org mailing lists have been disabled due to abuse. To subscribe to Kde-XXX please send an email to kde-XXX-request@kde.org with the subject "subscribe".


We have contacted upstream to see if we can solve it, but for now, bots vs humans: 1-0.

The second group of ‘spam sign-ups’ are not really spam sign-ups. They are done by people that want to post something on forum.kde.org, they don’t have any interest in any other service and will use their favorite nickname as full name because they care about their privacy. For us, it is hard to distinguish from the first group I described. We brainstormed a bit about possible solutions. Maybe just allow guest postings (postings without the need for registration), but that was discarded because it would raise the level of spam by automated bots. For now we don’t have a solution.

We also talked about some kind of unified user experience over all our services. Which might integrate some kind of service which users could use to promote their own apps. But the lack of resources will keep this idea away from the list with short term items.

In the afternoon we settled in the garden. Though it was a bit cold, it was very pleasant. We worked on setting up the Drupal-instance for the main-site. Ingo explained the structure we should use for the website. I dived in and will continue to work on it in the next period. I made Ingo happy by selling him my Asus Transformer, we played a bit with the dogs around us and then we left and went home.

This was a very productive meeting. Though it was a mini-sprint, it had great results. I’m regained some motivation, had a great time and have a reasonable idea about what we should focus on this year. I want to thank for Ingo for hanging out with me for 2 days, the linuxhotel.com for providing the best possible environment to work in and the eV for making it possible!

After that Ingo and me catched up on some work within the sysadmin and webteam. We started evaluating the main site. Last year at the WebWorld-meeting we decided that the new site should be revamped and we choose WordPress as CMS. Now a year later we still don’t have that site live. And it’s not going to happen. We simply don’t have the resources to get it up and running. What I suggested is that we stop trying to do it in WordPress and switch to Drupal. Changing the tool does not magically make the site, but what it does do is making it maintainable. The Dot is already Drupal and behindkde.org as well. That means the theme is finished and we can just focus on getting the content right. I’m personally motivated to get that new site live as soon as possible. If you want to help, drop a comment, if you don’t want to help, that’s fine, just don’t complain afterwards that links to the site are broken or content is missing or stuff like that.

For dinner we drove up to Dortmund where we ate at Mongos. You can get some pictures here. When we arrived we ended up in a crowed heading for some soccer match. And when we were done eating we … ended up in the same crowd leaving the stadium. Ah well.. dinner was good and we had a chance to brainstorm a lot. For example about bugs.kde.org. We really need to draw up a plan for that site. Each year we throw more iron at it to keep it running. But it is a beast. Basically we need to split it up into crashes, wishes and bugs and then we need to decide if bugzilla is the right choice for each of those categories. I’m betting it won’t be. We decided that we will work on a document which will layout the plans for the future of bugs.kde.org. Again if you want to be involved with drafting that document, leave a comment. If not… ah well, you know what i’m going to say.

Another big item we discussed was the future of subversion. Or rather, the lack of future for subversion. The sysadmin team is maintaining a complete infrastructure for subversion and for git. I think that’s silly and a waste of resources. Now a server is devided in two, one reviewboard for svn, one for git. Anonsvn and anongit servers. Commit hooks for svn and for git written in completely different languages. We are a big project, but we can’t maintain two. We need to shut down subversion at one point in time. It does not need to be tomorrow, it can be in a year or so, but it still needs a date to ever happen. We will draft a proposal up and sent it around for feedback.

Now we are back in the Conferenceroom, looking at the results of the UWG survey which holds quite some interesting results. Can’t wait to analyse it in more details and show the results to you all. More tomorrow!

Ingo is in love with my Android transformer, he already named it Eloise :). I probably wont see the device back this weekend or even after it. Leave a comment if you know a good price I can ask Ingo for it.

Signing off now, seems like breakfast is at 9 and Ingo wants to stick to it. Pfff….

Plans for this weekend involve pusing UWG, fixing the translation system of the wiki’s, do some web and sysadmin stuff. I’ll be more concrete tomorrow!

Today Ben Cooksley and I wrapped up the loose ends and put it online. It is a pleasure to invite you all to try it out!

You can now go to projects.kde.org. There you can login as usual. Then navigate to your pet projects. In the left side bar you will find a button to subscribe to the project. Press the ‘Apply’-button. There is no feedback when pressing that button (to be fixed later), but A few minutes later you will start to receive mails as soon as someone pushes commits to that project.

In a later stage we will probably add a possibility to subscribe only to a certain path, but that’s not to high on our current todo list. Also this is only for git repositories. You can still subscribe to SVN commits via http://commitfilter.kde.org/.

Try it out, leave a comment if you like this new service or not, give us feedback how to improve, or just use it and enjoy your new set of emails…