If you want to extract mails out of your .nsf database, this might be the tool you’re looking for. Bonus point: it’s written in Python !

Installing nlconverter and its dependencies

Here is how I installed nlconverter on a Windows 2000 (SP4) machine:

1. First I downloaded and installed the official Python builds for Windows (2.6.6 precisely):
   
   
   
   
   
2. Then Python for Windows extensions (build 214 for Python 2.6 in my case):
   
   
   
   
3. Finally I had to download the latest icalendar archive, then extract the \iCalendar-1.2\src\icalendar folder to C:\Python26\Lib\site-packages\:
   
4. Next step is to download nlconverter itself and extract it:
   

nlconverter GUI

First thing you have to do is to create an export of your mails as a .nsf database. Follow the previous link to get the instructions.

Now let’s convert this nsf to a mbox. nlconverter’s FAQ tells you to run the gui.exe program to perform the conversion.

Unfortunately it didn’t work for me:

So I tried the alternative approach by using the command line.

nlconverter command line

Again, most of the things I’m writing here are based on nlconverter’s FAQ:

1. First, we have to download the notes2mbox.py script from nlconverter’s mercurial repository, as this file is not distributed in the winnlc-alpha-1.zip archive I unzipped previously. Let’s put notes2mbox.py in C:\winnlc-alpha-1\:
   
2. Now we’ll modify the notes2mbox.py script to set the password (via the notesPasswd variable) and location (notesNsfPath variable) of the .nsf file. Here are the modifications I applied:

   --- notes2mbox.py.orig	2010-09-02 13:49:58.000000000 +0200
   +++ notes2mbox.py	2010-09-02 13:51:24.000000000 +0200
   @@ -14,8 +14,8 @@
    import NlconverterLib
   
    #Constantes
   -notesPasswd = "foobar"
   -notesNsfPath = "C:\\archive.nsf"
   +notesPasswd = "XXXXXXXXXXXXX"
   +notesNsfPath = "C:\\winnlc-alpha-1\\kevin-notes-big-backup-part-1.nsf"
   
    #Connection à Notes
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
   

3. Before running the script, we have to register a Notes DLL used by nlconverter:

   regsvr32 "C:\Program Files\Notes\nlsxbe.dll"
   

   
   And make the Python interpreter available system-wide:

   C:\winnlc-alpha-1>SET Path=%Path%;C:\Python26
   

4. Now we can run the notes2mbox.py script:

   C:\winnlc-alpha-1>C:\Python26\python.exe notes2mbox.py
   

If you’re lucky, you’ll get a nice mbox at the end of the process.

But I was not and the notes2mbox.py ended up with the following error:

Traceback (most recent call last):
  File "notes2mbox.py", line 21, in <module>
    db = NlconverterLib.getNotesDb(notesNsfPath, notesPasswd)
  File "C:\winnlc-alpha-1\NlconverterLib.py", line 43, in getNotesDb
    session = win32com.client.Dispatch(r'Lotus.NotesSession')
  File "C:\Python26\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
    dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 104, in _GetGoodDispatchAndUserName
    return (_GetGoodDispatch(IDispatch, clsctx), userName)
  File "C:\Python26\lib\site-packages\win32com\client\dynamic.py", line 84, in _GetGoodDispatch
    IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
pywintypes.com_error: (-2147221231, 'ClassFactory ne peut pas fournir la classe demand\xe9e', None, None)

As you can see, I tried hard to make nlconverter working, without any success. But this should not stop you to try. In fact I suspect the Lotus Notes installed on my machine to be crippled or corrupted (can’t really tell). So you may be more lucky than me. In any case, feel free to report any success or failure in the comment section below !

Error logging into server localhost: You must enable the Notes TCPIP port.

This appeared in the fat Lotus Notes client v7.0.3 running on Mac OS X Leopard (10.5).

To fix this issue, first of all, quit Notes. Then locate the Notes’ preference file attached to your current user. Mine was found in my home directory at /Users/kevin/Library/Preferences/Notes Preferences. At the end of this file, add these two directives:

TCPIP=TCP,0,15,0
Ports=TCPIP

Then relaunch Lotus Notes and switch from Island (Disconnected) location to another that will allow your client to listen to the network. In my case, Home (Network Dialup) did the trick:

Problem solved !

You can give the script a list of email headers to ignore while it compares mails between each others. This is particularly helpful to find duplicate mails having the exact same content but different UIDs.

I created this script to clean up a Maildir folder I messed up after moving repeatedly tons of mails from a Lotus Notes database. As you can see below, the same mail imported twice contain a variable header based on the date and time the import was performed:

This variable header make mails looks different from the point of view of the script. That’s explain why I implemented the HEADERS_TO_IGNORE parameter with the default set to X-MIMETrack.

The script is available on my GitHub repository. It was tested on MacOS X 10.6 with python 2.6.2 but should work on other systems and versions as the code is really simple (and stupid).

Here are some noisy photos I’ve taken with my Canon 7D and a 15-85mm f/3.5-5.6. If only I had a fast telephoto lens…

I used it to create some effects for my 4 el-cheapo Mac-Mah LED wider panels. This script helped me generate column and row presets of my 4×8 pixels LED matrix for some basic colors:

Here are some photos of my preliminary tests at home:

And finally photos of the panels on stage (taken by Toma Heroow during Cool Cavemen’s concert in last november):

As usual, use and hack this script at you own risks, and feel free to send me bug reports and contributions !

I’m no exception and the old and decrepit part of Cool Cavemen’s website still running on e107 was attacked. This was enough to crash my tiny server. Unfortunately this happened while I was on holidays. Without any time to address this issue properly, I decided to shutdown my web server. This explain why this blog and all Cool Cavemen’s websites were dead during half of july.

Now everything is back to normal (I hope), thanks to fail2ban. I created a set of rules (based on this article) to dynamically catch dDOS attempts and ban all IP addresses involved. Here is how I configured fail2ban…

First, create a new empty file at /etc/fail2ban/filter.d/apache-e107ddos.conf and put the following directives there:

# Fail2Ban configuration file
# Notes.:  Regexp to catch all attemps to exploit an e107 vulnerability.
# Author: Kevin Deldycke

[Definition]
failregex = <HOST>\s-\s-\s.*\s"(GET|POST).*\/(help_us|contact|config|avd_start|\*)\.php
            <HOST>\s-\s-\s.*(Casper|b3b4s|dex|Dex|kmccrew|plaNETWORK|sasqia|sledink|indocom) Bot Search
            <HOST>\s-\s-\s.*MaMa CaSpEr
            <HOST>\s-\s-\s.*rk q kangen
            <HOST>\s-\s-\s.*Mozilla\/4\.76 \[ru\] \(X11; U; SunOS 5\.7 sun4u\)
            <HOST>\s-\s-\s.*perl post
ignoreregex =

Then update you fail2ban config file (/etc/fail2ban/jail.local in my case) with the appropriate section:

[apache-e107ddos]
enabled  = true
filter   = apache-e107ddos
port     = http,https
action   = iptables-allports
logpath  = /var/log/apache*/*access.log
maxretry = 1

Then restart your fail2ban service:

$ /etc/init.d/fail2ban restart

And you’ll start to get those nice logs:

$ tail -F /var/log/fail2ban.log
2010-06-23 16:05:37,417 fail2ban.actions: WARNING [apache-e107ddos] Ban 193.33.21.199
2010-06-23 16:05:58,113 fail2ban.actions: WARNING [apache-e107ddos] Ban 89.108.116.226
2010-06-23 16:05:58,521 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.41.162.10
2010-06-23 16:05:58,541 fail2ban.actions: WARNING [apache-e107ddos] Ban 209.62.28.178
2010-06-23 16:06:03,573 fail2ban.actions: WARNING [apache-e107ddos] Ban 69.73.147.90
2010-06-23 16:06:42,975 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:44,227 fail2ban.actions: WARNING [apache-e107ddos] 69.41.162.10 already banned
2010-06-23 16:06:54,238 fail2ban.actions: WARNING [apache-e107ddos] 69.73.147.90 already banned
2010-06-23 16:07:50,305 fail2ban.actions: WARNING [apache-e107ddos] Ban 80.55.107.74

If screenshots were taken with a french version, instructions given here are for the english one. This will give you enough clues to perform the export whatever the localisation is. The Lotus Notes version I used was the 7.0.2 release.

So first, let’s start Notes and open your mailbox. You should be on a screen similar to this one:

Then, go to the File › Database › New Copy menu:

And you’ll get an export screen that’ll let you choose where to create a local copy of your database:

This will generate a .nsf file containing all your current mail.

Now that you have a backup, you are free to delete all your mails in Lotus Notes. By following this procedure regulary, you can create yearly or monthly archives of you mails without reaching the mailbox quota ! For example, this is how my local archive folder looks like:

This is a simple hack which get commit metadata out of a Subversion repository and generate an iCalendar file containing all commits of a given author. I used it back then to visualize in a calendar my commit activity. Nowadays this script is quite useless as services like Ohloh and GitHub provides great timeline and activity streams. But this script can still be usefull for private repositories.

And in the same spirit of this script, I uncovered maildir2ical.py, a script that look in a maildir folder for mails sent by a particular author, then generate an iCal file based on mail dates.

Then came Git. I knew that Git was powerful enough to let me manipulate the history (at my own risks). So I studied it during the last weeks until I found an acceptable way to do exactly what I had in mind. Here are my notes regarding this journey.

First, I need to get a local copy of my GitHub repository. That’s the place where I want all my code to reside at the end of the process.

cd ~
git clone git@github.com:kdeldycke/kev-code.git

In gitg, my untouched repository looks like this:

Notice all the pre-existing code.

Let’s create a history-injection branch from the init tag. The later is the root of my repository, as explained in my previous post on how I initialize my Git repositories.

git branch history-injection init

Then switch to our brand new branch:

git checkout history-injection

We are now in a safe and contained environment in which we can do all our dirty stuff. Let’s move the file we want to add in our repository:

cp ~/scripts/website-backup-2006_04_30.py ~/kev-code/website-backup.py

Commit this new file locally, as usual, but with a commit date set in the past:

cd ~/kev-code
git add --all
git commit --all --date="2006-04-30 23:17" -m "First version of a script to backup several remote websites via FTP and make bzip2 archives."

I can repeat the last steps to reconstruct the commit history of my website-backup.py script:

cp ~/scripts/website-backup-2006_10_29.py ~/kev-code/website-backup.py
git commit --all --date="2006-10-29 23:13" -m "Delete previous backups if nothing has changed."
cp ~/scripts/website-backup-2006_11_01.py ~/kev-code/website-backup.py
git commit --all --date="2006-11-01 23:14" -m "Keep monthly bzip2 snapshots of backups and incremental backups of the last 32 days thanks to rdiff-backup."
(...)

At last, the history-injection branch contain all version of website-backup.py:

Now I’ll use the rebase directive to insert the history-injection branch back in the main line (aka master). This insertion will take place just after the init tag. This translates to the following Git command:

git rebase --preserve-merges --onto history-injection init master

The --preserve-merges option is really important here to not let Git takes too much initiatives. Without this option, all our banches between the init tag and the head of the master branch will be rebased. Believe me, that’s not what we want.

I no longer need my temporary history-injection branch. Let’s remove it:

git branch -D history-injection

Now you should have a unique and straight history line from init tag to master head. Like this:

Commits appears to be ordered as they should but you may not be as lucky as me. In fact the recently merge commits are stuck at the “bottom” (just after the init tag, as we asked Git to do on rebase). And you may find you in a situation where commits of the whole master branch are not chronologically ordered.

Here is such an example. It happened when I tried to rebase the full history of my system-backup.py script:

I haven’t found a way to tell Git how to rebase by following commit dates. I know that something can be done with a command like:

git rebase --interactive init

But I haven’t succeeded yet. So I left these commits unsorted for now. I may write another blog post in the future if I find a way to cleanly sort them. In the mean time, If you have a solution, I’ll be happy to ear that !

Finally, when we have something that looks good, we can push our changes to our remote GitHub repository:

git push origin

But Git will complain: changing already-pushed commits is bad. As I explained several weeks ago, it’s dangerous but I don’t care. I’m the only user of this repository. So let’s bypass Git’s wise warnings:

git push origin +master:master

Et voilà ! By repeating these steps several times, I moved my code to GitHub, with a consistent and clean commit history.