Approach for Pipermail

(Sorry for the gap in posts, I just got back from a weekend vacation without internat access)

"pickle" refers to the Python standard object serialization library cPickle
"pipermail" refers to one of the default pluggable archivers packaged with Mailman 3

I believe that the simplest way to approach this (at least for right now) is to take not of all existing functionality and to write a class that pipermail can use instead of pickle. For example, the following is in Archiving/pipermail.py:

self.articleIndex[article.msgid] = pickle.dumps(article)

In order to replicate this behavior, I will write a similar method that would provide the same functionality while interfacing with a SQL database via the STORM API instead of pickle's object serialization. See #1 of To Do for details

To Do for Archiver/pipermail.py:
1. The method store_article in the Database class, which defines the basic sorting logic for a database, uses the cPickle.dumps() method to return the article as a string representation instead of writing it to a file. This returned string is then stored in a dictionary instance variable (articleIndex) with its message id as the key. When implementing the STORM conversion, there needs to be a mechanism for generating and returning the article's message id for this same purpose.

2. The __init__(...) method of class T (Python formatter class) attempts (in a try-except block) to reload the previously pickled data and creates a new directory of no pickled data exists. However, according to the comment documentation within the method the code is only run for legacy lists and this functionality is replicated within Archiver.py. Archiver.py in turn contains doctest that states that archives such as pipermail should be pointed to the correct directories in order to perform the archiving, so for now I'm going to ignore this second part and rewrite the pickle functionality in T's __init__ method for legacy lists.

3. The T class's close() method "Close[s] an archive, save[s] its state, and update[s] any changed archives." To accomplish this, it calls pickle.dump() and passes its current state (the namespace content for the object) and the file object opened to the externally-defined based directory and pipermail pickle. This method pickles the archive state by calling pickle.dump() and passing a path to the pickle file and a reference to the __dict__ property (which contains the namespace contents of the object). In this method, I will map the pipermail schema from the pickle instance to a SQL schema instead and write to a database file instead of a .pck file.

4. The only other call to a pickle method in Archiver/pipermail.py is in the class BSDDBdatabase, which is an extension to the Database class and further defines database sorting logic. In its getArticle() method, it attempts to load pickle data from the pickle representation in self.articleIndex[msgid], which was previously created in the store_article method (inherited from Database - see #1). In this method in my STORM class, I'll need to perform the reverse of what I do in number 1 and load the article with the proper articleIndex from the database.

As of now, the above plan describes what I envision implementing to replace pickling with SQL for archiving messages. I'm going to consult with Barry and Terri (I've been meaning to contact her) to determine whether or not this is a valid course of action or whether it will not suit the needs of the default archiving system.

TDD

So a couple of days ago I decided that it was in my best interest to really figure out how Test Driven Development (or TDD) is useful and then to get some hands-on experience. To do so, I read the chapter on Unit Testing in the book Dive Into Python, written by Mark Pilgrim. I wanted to get a more generic sense of how TDD works without getting bogged down by minor details and intricacies of a larger testing suite such as Mailman's.

To employ this newly found knowledge, I wrote a quick test class for one of the classes that runs my server and it turned out to be incredibly helpful. Needless to say, in between my Mailman development I'm going to continue developing a comprehensive test suite for my server in order to both hone my TDD skills and to finally get the damn thing to be as perfectly stable as I'd like.

All in all, I'd say that was a couple of days well spent now that I have a significantly better understanding of TDD and its practical uses. I'm going to dig through the Mailman code and see what I can find out about how the test suite is run and how I should go about defining the test cases for my pipermail Storm conversion.

Project Details

Alright so after (admittedly) reading Mailman documentation on and off, usually alternating days for another project and my other job, I finally decided it was time to talk to Barry (mentor/project leader) about the scope of my project and what I thought I needed to do. I was kind of close, but nowhere near close enough to what I should have been (and expect of myself). Since I've never dealt with a project of this scale before, I'm still trying to figure out how best to approach understanding the thousands of lines of code contained in this project. Granted I'm only technically supposed to be working on the Pipermail Storm implementation, but I hate not knowing what's going on in the rest of the project.

Going back and rereading my proposal has been helpful in keeping me focused while analyzing the archiving code and other documentation since it's easy to lose sight of what you're trying to do. Python also isn't my language of choice so I'm working on figuring out a lot of the tricks I see employed throughout the code. But this was my goal - throwing oneself into a new environment is generally the best way to learn. I'm not totally up the creek though, I do have enough experience in Python to get by most code without little or no problem since it's the development language of my iPhone application's server.

As far as further project specificity goes, I learned that Mailman 3 has an interface that defines an API between the mailman core and what we're calling an "archiver" instead of a few different implementations of archivers all using pipermail at their core. Mailman 3 comes with several implementations of this interface with varying degrees of scope, for example pipermail is a more comprehensive (Barry used "intimate" to describe it) archiver while mail-archive.com just forwards the message to that service (Which, as the website states, "turns your mailing list into a searchable archive"). After a bit more research, I've determined that Hyperdatabase and Archiver/pipermail need to be rewritten to use the Storm API rather than cPickle and to eliminate some generalizations that are not necessary any more. Bottom line there: anything using pickle needs to be rewritten to support storm (with some other schema modifications).

After I do that, i'll have to start working on the upgrade script to migrate Mm2 pickle data to this new schema. I'm confident but dear God I hope I don't screw this up