If you happen to be looking for my GSoC blog, I moved to wordpress ages ago and forgot to post about it here.
New link: http://dcrodman.wordpress.com/
Wednesday, July 27, 2011
Tuesday, May 31, 2011
Approach for Pipermail
(Sorry for the gap in posts, I just got back from a weekend vacation without internat access)
"pickle" refers to the Python standard object serialization library cPickle
"pipermail" refers to one of the default pluggable archivers packaged with Mailman 3
I believe that the simplest way to approach this (at least for right now) is to take not of all existing functionality and to write a class that pipermail can use instead of pickle. For example, the following is in Archiving/pipermail.py:
To Do for Archiver/pipermail.py:
1. The method store_article in the Database class, which defines the basic sorting logic for a database, uses the cPickle.dumps() method to return the article as a string representation instead of writing it to a file. This returned string is then stored in a dictionary instance variable (articleIndex) with its message id as the key. When implementing the STORM conversion, there needs to be a mechanism for generating and returning the article's message id for this same purpose.
2. The __init__(...) method of class T (Python formatter class) attempts (in a try-except block) to reload the previously pickled data and creates a new directory of no pickled data exists. However, according to the comment documentation within the method the code is only run for legacy lists and this functionality is replicated within Archiver.py. Archiver.py in turn contains doctest that states that archives such as pipermail should be pointed to the correct directories in order to perform the archiving, so for now I'm going to ignore this second part and rewrite the pickle functionality in T's __init__ method for legacy lists.
3. The T class's close() method "Close[s] an archive, save[s] its state, and update[s] any changed archives." To accomplish this, it calls pickle.dump() and passes its current state (the namespace content for the object) and the file object opened to the externally-defined based directory and pipermail pickle. This method pickles the archive state by calling pickle.dump() and passing a path to the pickle file and a reference to the __dict__ property (which contains the namespace contents of the object). In this method, I will map the pipermail schema from the pickle instance to a SQL schema instead and write to a database file instead of a .pck file.
4. The only other call to a pickle method in Archiver/pipermail.py is in the class BSDDBdatabase, which is an extension to the Database class and further defines database sorting logic. In its getArticle() method, it attempts to load pickle data from the pickle representation in self.articleIndex[msgid], which was previously created in the store_article method (inherited from Database - see #1). In this method in my STORM class, I'll need to perform the reverse of what I do in number 1 and load the article with the proper articleIndex from the database.
As of now, the above plan describes what I envision implementing to replace pickling with SQL for archiving messages. I'm going to consult with Barry and Terri (I've been meaning to contact her) to determine whether or not this is a valid course of action or whether it will not suit the needs of the default archiving system.
"pickle" refers to the Python standard object serialization library cPickle
"pipermail" refers to one of the default pluggable archivers packaged with Mailman 3
I believe that the simplest way to approach this (at least for right now) is to take not of all existing functionality and to write a class that pipermail can use instead of pickle. For example, the following is in Archiving/pipermail.py:
self.articleIndex[article.msgid] = pickle.dumps(article)In order to replicate this behavior, I will write a similar method that would provide the same functionality while interfacing with a SQL database via the STORM API instead of pickle's object serialization. See #1 of To Do for details
To Do for Archiver/pipermail.py:
1. The method store_article in the Database class, which defines the basic sorting logic for a database, uses the cPickle.dumps() method to return the article as a string representation instead of writing it to a file. This returned string is then stored in a dictionary instance variable (articleIndex) with its message id as the key. When implementing the STORM conversion, there needs to be a mechanism for generating and returning the article's message id for this same purpose.
2. The __init__(...) method of class T (Python formatter class) attempts (in a try-except block) to reload the previously pickled data and creates a new directory of no pickled data exists. However, according to the comment documentation within the method the code is only run for legacy lists and this functionality is replicated within Archiver.py. Archiver.py in turn contains doctest that states that archives such as pipermail should be pointed to the correct directories in order to perform the archiving, so for now I'm going to ignore this second part and rewrite the pickle functionality in T's __init__ method for legacy lists.
3. The T class's close() method "Close[s] an archive, save[s] its state, and update[s] any changed archives." To accomplish this, it calls pickle.dump() and passes its current state (the namespace content for the object) and the file object opened to the externally-defined based directory and pipermail pickle. This method pickles the archive state by calling pickle.dump() and passing a path to the pickle file and a reference to the __dict__ property (which contains the namespace contents of the object). In this method, I will map the pipermail schema from the pickle instance to a SQL schema instead and write to a database file instead of a .pck file.
4. The only other call to a pickle method in Archiver/pipermail.py is in the class BSDDBdatabase, which is an extension to the Database class and further defines database sorting logic. In its getArticle() method, it attempts to load pickle data from the pickle representation in self.articleIndex[msgid], which was previously created in the store_article method (inherited from Database - see #1). In this method in my STORM class, I'll need to perform the reverse of what I do in number 1 and load the article with the proper articleIndex from the database.
As of now, the above plan describes what I envision implementing to replace pickling with SQL for archiving messages. I'm going to consult with Barry and Terri (I've been meaning to contact her) to determine whether or not this is a valid course of action or whether it will not suit the needs of the default archiving system.
Wednesday, May 25, 2011
TDD
So a couple of days ago I decided that it was in my best interest to really figure out how Test Driven Development (or TDD) is useful and then to get some hands-on experience. To do so, I read the chapter on Unit Testing in the book Dive Into Python, written by Mark Pilgrim. I wanted to get a more generic sense of how TDD works without getting bogged down by minor details and intricacies of a larger testing suite such as Mailman's.
To employ this newly found knowledge, I wrote a quick test class for one of the classes that runs my server and it turned out to be incredibly helpful. Needless to say, in between my Mailman development I'm going to continue developing a comprehensive test suite for my server in order to both hone my TDD skills and to finally get the damn thing to be as perfectly stable as I'd like.
All in all, I'd say that was a couple of days well spent now that I have a significantly better understanding of TDD and its practical uses. I'm going to dig through the Mailman code and see what I can find out about how the test suite is run and how I should go about defining the test cases for my pipermail Storm conversion.
To employ this newly found knowledge, I wrote a quick test class for one of the classes that runs my server and it turned out to be incredibly helpful. Needless to say, in between my Mailman development I'm going to continue developing a comprehensive test suite for my server in order to both hone my TDD skills and to finally get the damn thing to be as perfectly stable as I'd like.
All in all, I'd say that was a couple of days well spent now that I have a significantly better understanding of TDD and its practical uses. I'm going to dig through the Mailman code and see what I can find out about how the test suite is run and how I should go about defining the test cases for my pipermail Storm conversion.
Monday, May 23, 2011
Project Details
Alright so after (admittedly) reading Mailman documentation on and off, usually alternating days for another project and my other job, I finally decided it was time to talk to Barry (mentor/project leader) about the scope of my project and what I thought I needed to do. I was kind of close, but nowhere near close enough to what I should have been (and expect of myself). Since I've never dealt with a project of this scale before, I'm still trying to figure out how best to approach understanding the thousands of lines of code contained in this project. Granted I'm only technically supposed to be working on the Pipermail Storm implementation, but I hate not knowing what's going on in the rest of the project.
Going back and rereading my proposal has been helpful in keeping me focused while analyzing the archiving code and other documentation since it's easy to lose sight of what you're trying to do. Python also isn't my language of choice so I'm working on figuring out a lot of the tricks I see employed throughout the code. But this was my goal - throwing oneself into a new environment is generally the best way to learn. I'm not totally up the creek though, I do have enough experience in Python to get by most code without little or no problem since it's the development language of my iPhone application's server.
As far as further project specificity goes, I learned that Mailman 3 has an interface that defines an API between the mailman core and what we're calling an "archiver" instead of a few different implementations of archivers all using pipermail at their core. Mailman 3 comes with several implementations of this interface with varying degrees of scope, for example pipermail is a more comprehensive (Barry used "intimate" to describe it) archiver while mail-archive.com just forwards the message to that service (Which, as the website states, "turns your mailing list into a searchable archive"). After a bit more research, I've determined that Hyperdatabase and Archiver/pipermail need to be rewritten to use the Storm API rather than cPickle and to eliminate some generalizations that are not necessary any more. Bottom line there: anything using pickle needs to be rewritten to support storm (with some other schema modifications).
After I do that, i'll have to start working on the upgrade script to migrate Mm2 pickle data to this new schema. I'm confident but dear God I hope I don't screw this up
Going back and rereading my proposal has been helpful in keeping me focused while analyzing the archiving code and other documentation since it's easy to lose sight of what you're trying to do. Python also isn't my language of choice so I'm working on figuring out a lot of the tricks I see employed throughout the code. But this was my goal - throwing oneself into a new environment is generally the best way to learn. I'm not totally up the creek though, I do have enough experience in Python to get by most code without little or no problem since it's the development language of my iPhone application's server.
As far as further project specificity goes, I learned that Mailman 3 has an interface that defines an API between the mailman core and what we're calling an "archiver" instead of a few different implementations of archivers all using pipermail at their core. Mailman 3 comes with several implementations of this interface with varying degrees of scope, for example pipermail is a more comprehensive (Barry used "intimate" to describe it) archiver while mail-archive.com just forwards the message to that service (Which, as the website states, "turns your mailing list into a searchable archive"). After a bit more research, I've determined that Hyperdatabase and Archiver/pipermail need to be rewritten to use the Storm API rather than cPickle and to eliminate some generalizations that are not necessary any more. Bottom line there: anything using pickle needs to be rewritten to support storm (with some other schema modifications).
After I do that, i'll have to start working on the upgrade script to migrate Mm2 pickle data to this new schema. I'm confident but dear God I hope I don't screw this up
Monday, April 25, 2011
Public Patch
Alright so for this project I found out that I needed to submit a public patch along with my GSoC application. Needless to say, I freaked out and went digging through the Bugs page of the Mailman page on Launchpad trying to find something that I could do without having to figure out the intricacies of the software. Yep, it didn't work. Thankfully Barry had mercy on me and gave me a project that involved adding support for deleting domains to the REST API.
Of course this discovery came in the middle of the last few weeks of my freshman year at College of Charleston, so I was busy beyond belief and was desperately looking for random time slots in which I could work on this project. I've never worked on open source software before, so the size of the source code was daunting and I began to struggle as I worked through several files line by line trying to figure out how everything works. After looking at lists.py for comparison, I was unsure as to whether I should be writing an entire module to support deleting of domains or writing a method in IDomainManager to perform the action. Barry came to the rescue again and led me to realize that the method I had (correctly!) intended to implement was already implemented by a model class...and that I had been looking at the interface definition for a DomainManager without realizing. Go figure.
Finally I got my hands on the relevant files to the project and after about an hour and a half, I had written the extension to the REST API that allowed for deleting domains through HTTP DELETE connections. After pestering Barry again, I found out that Mailman is developed using Test Driven Development philosophy; something I only knew about in theory. Now I knew that as soon as I figured it out I'd take it and run with it, so I jumped right into writing tests and, you guessed it, failed horribly. I really, really don't like it when I don't understand CS concepts, so over a span of about three days (random intervals due to school work) I worked to figure out how to build and run tests. I was confident that my delete method in domains.py would work, so I began to get incredibly frustrated that I couldn't get the tests to pass.
This morning (April 25th), I gave in and ping'd Barry again on IRC and with his patient guidance and my brute determination, I managed to push what I hope is a correct solution to the problem via Launchpad.
This whole process was a fantastic way for me to get to work with the software and become familiar with at least part of the functionality. It also gave me a chance to work with test driven development and Bazaar for version control (again, something I only knew about in theory) using tutorial I found to push my branch to Launchpad.
I'm pumped to get started on Mailman for the summer, it's going to be cool to dig into the documentation and figure out how everything works so that I can carry out my proposal as planned. In the meantime, I'm finishing up a little bit of development on my iPhone application before the artwork gets finished so I can push it to the App Store. It's going to be busy summer!
Of course this discovery came in the middle of the last few weeks of my freshman year at College of Charleston, so I was busy beyond belief and was desperately looking for random time slots in which I could work on this project. I've never worked on open source software before, so the size of the source code was daunting and I began to struggle as I worked through several files line by line trying to figure out how everything works. After looking at lists.py for comparison, I was unsure as to whether I should be writing an entire module to support deleting of domains or writing a method in IDomainManager to perform the action. Barry came to the rescue again and led me to realize that the method I had (correctly!) intended to implement was already implemented by a model class...and that I had been looking at the interface definition for a DomainManager without realizing. Go figure.
Finally I got my hands on the relevant files to the project and after about an hour and a half, I had written the extension to the REST API that allowed for deleting domains through HTTP DELETE connections. After pestering Barry again, I found out that Mailman is developed using Test Driven Development philosophy; something I only knew about in theory. Now I knew that as soon as I figured it out I'd take it and run with it, so I jumped right into writing tests and, you guessed it, failed horribly. I really, really don't like it when I don't understand CS concepts, so over a span of about three days (random intervals due to school work) I worked to figure out how to build and run tests. I was confident that my delete method in domains.py would work, so I began to get incredibly frustrated that I couldn't get the tests to pass.
This morning (April 25th), I gave in and ping'd Barry again on IRC and with his patient guidance and my brute determination, I managed to push what I hope is a correct solution to the problem via Launchpad.
This whole process was a fantastic way for me to get to work with the software and become familiar with at least part of the functionality. It also gave me a chance to work with test driven development and Bazaar for version control (again, something I only knew about in theory) using tutorial I found to push my branch to Launchpad.
I'm pumped to get started on Mailman for the summer, it's going to be cool to dig into the documentation and figure out how everything works so that I can carry out my proposal as planned. In the meantime, I'm finishing up a little bit of development on my iPhone application before the artwork gets finished so I can push it to the App Store. It's going to be busy summer!
Thursday, April 7, 2011
Project Proposal Version One
Mailman 3 Pipermail SQL Implementation and Mailman 2 Archive Uprade Script.
Mailman 2's Pipermail archiver is currently built on a persistence layer of
Python pickles, which is memory and performance inefficient. The end result
of this project will be to rewrite the Mailman 3 Pipermail implementation to
be backed by SQL, using the Storm (storm.canoincal.com) ORM. This will be
achieved by mapping the MM2 pickle schema to a new MM3 SQL schema, along with
an upgrade script to convert existing Pipermail archives to the new format.
This upgrade script could be run standalone, or as part of a general MM2->MM3
upgrade procedure. APIs will be added and updated as needed to support both
the on-the-fly addition of new messages, as well as full archive
regeneration. Upon completion of this objective, I will work on the Stable
will be kept for the old URLs so that they can be redirected to the new stable
URLs, ensuring that current links will not be broken.
Python pickles, which is memory and performance inefficient. The end result
of this project will be to rewrite the Mailman 3 Pipermail implementation to
be backed by SQL, using the Storm (storm.canoincal.com) ORM. This will be
achieved by mapping the MM2 pickle schema to a new MM3 SQL schema, along with
an upgrade script to convert existing Pipermail archives to the new format.
This upgrade script could be run standalone, or as part of a general MM2->MM3
upgrade procedure. APIs will be added and updated as needed to support both
the on-the-fly addition of new messages, as well as full archive
regeneration. Upon completion of this objective, I will work on the Stable
URL project proposal by using the Message-ID hashes as keys for generating
stable URLs to the entries in the database. As an added feature, a mappingwill be kept for the old URLs so that they can be redirected to the new stable
URLs, ensuring that current links will not be broken.
Milestones
These are initial goals, students may work with their mentor over the Summer to update these goals as needed.
Start of Program (May 24)
My goal before I start is to spend time from now until the start of the actual coding to figuring out how the entire Mailman system works and how my Pipermail implementation needs to fit into the system.
Midterm Evaluation (July 12)
Deliverables:
- Nearly complete if not fully working implementation of Pipermail using the Storm ORM API
- Nearly complete if not fully working upgrade script to move through the Mailman 2 archives and convert the Python Pickle Data to a SQL database using Storm
Final Evaluation (Aug 16)
List your proposed deliverables by the end of the Summer:
- Fully working implementation of the Pipermail backend using the Storm ORM API
- Fully working upgrade script to convert Mailman 2 Pipermail archives from Pickle scheme to the Mailman 3 SQL scheme
- Stable URL generation for entries in the Mm3 SQL database using the Message-ID hashes as keys with support for mapping old URLs to the new stable ones
Subscribe to:
Posts (Atom)