Nevertheless, I will try to move the development forward a bit, albeit not with the rigurosity and elegance of George’s software design. Code like hell activated folks! I built a “pda_mess” branch where I’ll start my work.

The goal: To have phpBB3< ->mail working asap. Hopefully later on George will be able to clean up the mess and integrate the development into the main “trunk” branch.

Yesterday I updated some of the code to support php 5.3.0. Also updated Zend framework. It seems that php is becoming increasingly strict with semantics, also there are several functions that are becoming deprecated. split() is giving me a headache right now, it’s beign replaced in favor of preg_split() is more powerful and faster. Sadly, split() is being used in the swift email library and in simpletest as well. All right, time to update the rest of the libraries…

.pd

As you already know, in these open source projects sometimes real life gets in the way, but after almost 9 years of development we can say with confidence that we always return to m2f!

Well, m2f 2.0 is a complex design-focused message hub. Email handling complexity by itself is a big deal, and that is only one (albeit one of the most important) of the channels that m2f 2.0 supports. So, prioritizing is an important part of our development plans, remember we have only one senior developer (George) and one junior developer (me) right now. If you need m2f and can give us a hand (ie. 2-5 hours a week) then be sure to drop us a line - pm GeorgeC or Pda0 on the forum.

The current state of m2f 2.0 already allows for basic chaining of email, phpbb2, phpbb3, feed and other minor channels. Messages get threaded and attachments get processed, in a basic level (For more info read about MIME handling on previous blog posts). In general, the core m2f functionality is there, even though still in a immature state. One of the fun things that you can already do, and wasn’t possible with m2f 1.x, is binding phpbb2 and phpbb3 together (Or two phpbb3 installations together).

So, what’s next. There so many tasks lurking on the tracker, that we need some strategy to slowly crunch each of them. At the moment we have three objectives that drives our development priorities:

1. The requirement of one of our sponsors that needs m2f working with mailman+phpbb3 asap.
2. Build a very simple m2f beta-quality solution involving only some of the functionality.
3. Reorganize milestones from functionality classes to integral and incremental functionality releases.

Point 1 mainly focuses in mailman<->phpbb3 integration.

In point 2, we are building a feed->phpbb3 solution that can be released as a separate, beta-quality MOD release. Even though this task will divert us a bit from the main line of development, it will help us get a bigger userbase of users/testers and also prove the new concepts in production environments.

Point 3 deserves some explanation. If you look at the milestones in the tracker, you will see that requirements are grouped thinking in the types of functionality that are needed. For example “Email features” or “messagers & filters”, and so on. This helped the development a bit, as developers can focuse in one type of problem at a time, but it also greatly reduced the integral functionality of a m2f release. We want to change that strategy so that every m2f release has a bit of functionality from each class, and thus is more practical for normal use.

Other than these, we’ve been working in finishing Feed channel and other general design improvements. In latest SVN, Feed no longer (re)imports feed posts that were already imported in previous chain runs. Also there are big improvements in the html->bbcode parser.

Stay tuned!

.pd

As I’ve been doing a huge amount of work on the email processing feature of m2f 2.0, I’d like some assistance with testing.

The new functionality is in the Email Import channel. In short, it will convert a multipart email (i.e. one in plain text as well as HTML, or an email with attachments) into a single, flat message suitable for using later in an m2f chain.

For the present, all attachments are stripped from the incoming message, and saved into a directory on your filesystem. There’s a new File Manager class which servers up the files, given a unique URL. For example, on my test server, an attachment will be given a URL like this:

http://m2f/download?file=5863b365ee66b7a28546b4f07e09913b.

If that URL is entered into a browser window, m2f will offer the file for download.

I’d like you to test the following functionality:

• check that all processed emails are legible, and well-formatted
• check that all attachments are correctly stripped, and are downloadable from the specified URL

Testing the MIME handling

There is a specific sequence of operations you must follow to properly test the new functionality. Please follow the instructions carefully. You will need an archive of old emails to test with, in mbox format. Please try to test at least some with HTML parts, and attachments, where possible. This is the main focus of the test!

1. Work with a copy of your mbox file, as we will be deleting messages individually from it as the test progresses.
2. First, it’s imperative that you do a new checkout from SVN (the MIME parsing feature isn’t yet available in a release), and run the entire test suite. Any errors must be reported and resolved before there’s any point in proceeding.
3. Set up the following chain using the m2f Admin Control Panel:
   • Import channel: M2f_Module_Channel_Email_Import_Mbox
     • set filePath to the path to your mbox file
     • set maxMsgs to 1
     • set deleteMsgs to false (i.e. do not check the box)
   • Export channel: M2f_Module_Channel_FileWriter
     • set the filePath to the path of a file which is accessible using your webserver, with the suffix “.html”. For my tests, I use a file which can be accessed using the URL http://localhost/mime_test.html.
4. Run the chain. This should result in the contents of the first email in the mbox being parsed by m2f, and written to the file specified in the Filewriter export.
5. View the HTML file in your browser. If the original email contained any HTML, the generated file should be in HTML as well, and should already look good in your browser. If the original was plain text only, the contents of the HTML file will actually be plain text, so you’ll need to View Source in your browser to see the correct output. Please ignore the fact that a plain text message looks crap when viewed as HTML - that’s expected behaviour!
6. Check that the message looks good - it should be legible, with no missing images or text. Please note that it will not necessarily look the same as in your email client, as at present I’m stripping all CSS from the HTML headers. This may change in future.
7. Check that any attachments were correctly scrubbed, and that they are now accessible using the given URL. A block of text like this:
   An attachment was scrubbed:
Name: my_pic.jpg
Type: image/jpeg
URL : http://m2f/download?file=98070067013836d34358de6c092ef3c0should have been added to the bottom of the message for each attachment. Copy the URL, paste it into your browser, and you should get the option to download the file. Check the file is of the correct type (i.e. Microsoft Word documents open OK in Word, etc.), and that the contents is as expected.
8. If anything went wrong with the parsing - if the email looks weird, images are missing, attachment are not accessible, etc. - please do the following:
   • copy the RAW source of the email in question from your mbox file. It will be the  text starting from line 1, and going up to the next instance of this string: a new line, followed by the word “From”, followed by a space.
   • paste the raw email into a new text file. Please use the same encoding as the mbox file - if in doubt, use UTF-8.
   • replace any sensitive data with dummy data - but please don’t in any way alter the syntax of the email
   • email the file as an attachment to me at george@mail2forum.com, explaining what the problems were.
9. If the output looks OK, delete the raw email from the mbox (see above for how to identify a single email). This should leave the first line of the mbox file as the “From …” line of email number 2.
10. Run the chain again for the second email.

I’m very grateful for your help with testing this complex bit of code!

After the initial importing of a message, m2f analyzes the message content in order to understand its structure. If the message is being imported from an email channel, that means looking the current format of the message (ie. html, plain text and so on), checking if the email has attachments (normal attachments or inline attachments), etc. The way that emails handle multiple formats and attachments is called MIME Handling. As quoted from wikipedia:

Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of e-mail to support:

• Text in character sets other than ASCII
• Non-text attachments
• Message bodies with multiple parts
• Header information in non-ASCII character sets

Yes… daily life email messages are more complex than you could think! A typical email could have several parts, including a plain-text AND html-version of the message, inline images, file attachments and so on. m2f goes through all the parts and cleverly guesses which parts should be used for the message conversion process, which should be stored for later use, and which we’ll actually discard as message scrap.

m2f is being designed to alter the original message as well, which is an important matter for security’s and flexibility’s sake. For example, it will be able to sanitize html messages in order to try and filter out common code-inyection vulnerabilities (such as Cross Site Scripting). This will be configurable in the future.

Right now m2f mainly passes html as-is to the export channel, but first it strips all attachments and stores them in the m2f attachments directory. It then adds a link to every attachment at the bottom of the message, that points to the attachment that was stripped. This might be early attachment management, but it is working very well! Check this out:

Original Outlook Email

m2f Processed Email

Did you notice the “scrubbed attachment” on the Processed Email? Of course this is bare-bones implementation, there is still a lot to do, but the most difficult part (MIME Handling!) is almost over. The screenshots above are a result of a “mbox->filewriter” chain configured with current SVN

.pd

These have been busy days! George is finishing off MIME handling of email messages and building the general architecture that will eventually support full attachment management between channels.

I have been working in the Feed class, adding export functionality and also did some heavy refactoring to get cleaner code. There are still things to be done, in particular the configuration of the channel in the Admin-Cp, but we are almost there.

I would like to tell you a story about what’s involved in building a channel with the current state of the code (alpha 4):

• First of all, as always, you have to draw a basic sketch of what you want to do. For Feed I knew I wanted to import web feeds (mainly rss) into a forum, and export, say, latest 10 posts in a given forum out to a rss xml file.
• It is important to think whether your channel will only import, export, or both!
• Then, you build the framework of your channel, classes and interfaces, in the “lib\M2f\Module\Channel” directory. The tests will go into “\tests\lib\M2f\Module\Channel” directory.
• Then you can start building up the tests, then adding code to make them pass. You always build the test before writing actual code because it cleans up the design and also helps developing very robust code!

m2f channels revolve around objects called “Generic Messages”, which are, in essence, a data structure that holds all the information a message could eventually have. The channel converts messages from other sources such as Forums and Email into a Generic Message, or a Generic Message into a post or email message.

Having a Generic Message structure is the central approach of how m2f bind channels together. Instead of building a n-to-n format converter between channels, we decided using an intermediate format that every channel must know about.

So, for import channels, the difficulty resides in connecting to the remote software (via database link, POP, propietary API… you name it) and extracting messages in the original format. Then, convert that format into a Generic Message that can be further processed.

For export channels, the algorithm must take a Generic Message, convert it to the destination format, and finally connect to the remote software to submit the new message.

This might sound easy enough, but there are several pitfalls, including: Message threading, ugly code at the remote software side, attachment handling, weird uses cases (what happens if….?) and others. So, you have to take these into consideration if you want to develop a robust channel.

.pd

Installation & Basic Chain

The video shows the basic installation procedure, and setting up a test chain with a Random Text Generator input and a File Writer output.

m2f 2.0 Alpha 4: mbox email to phpBB 3

An introduction to m2f v2.0 alpha 4, released Feb 11th, 2009. The video shows the connection procedure with a remote host such as phpBB, and setting up a chain from a mbox email file to a phpBB 3 forum.

The m2f development team are delighted to announce the immediate availability of Alpha release 2.0.0a4. This is an alpha-quality build of the forthcoming v2.0 release, which will see the transformation of m2f from a simple phpBB -> email link into a multi-faceted message hub.

This release is aimed at alpha/beta-testers only. It is absolutely not ready for production use. Installation requires a long and precise list of instructions, and testing requires a number of dedicated resources, with potential risk to databases. Support will not be offered unless accompanied by bug reports on our Issue Tracker. Please only download this if you are a member of the Beta Testers group.

For an indication of the new features in this release, please see the roadmap on the m2f Issue Tracker. For an introduction to v2.0, help with installation, testing and setup, please see our documentation.

The release can be downloaded from Sourceforge at the following link:
https://sourceforge.net/project/showfiles.php?group_id=68964&package_id=67748&release_id=660489

If there are any PHP developers who would like to be involved in the development of m2f v2.0, please get in touch with Pda0 or Georgec on the forum!

Enjoy!

Beta testers: You’ll have work to do pretty soon! We submitted a new page into our documentation : v2.0 troubleshooting. This page includes bug reporting, testing environment resetting, etc. If you are a beta-tester (and thus member of the beta testers group in the forum), be sure to read that guide and comment!

Oh, and by the way, for sanity’s sake George has improved the configuration procedure for phpBB installations - no more host-key copying&pasting! You can review the updated install instructions on the m2f installation page.

Other than the occasional bug-squashing, we have encountered some interesting design decisions. There is a lot yet to be implemented, such as message routers, which will provide branching and probably chain-chaining (did that made sense?), among other things, but it is amazing how basic and plain functionality can make our developer minds go berserk! Just go and read some bug reports to see what I’m talking about.

On the planning side, now that we are near releasing the Alpha 4 build, we will start doing some heavy reorganization on the features present on future releases. We’ll try to release more often and thus with less new features per build. Note that even though this is an alpha-quality release, which means that there is a quite long feature list pending development, this is by no means a small or buggy release. Each time we release a m2f build, we will do our best to provide users of bug-free software, and with each additional release we plan to add at least one major feature until we finish the first Release Candidate.

.pd

• Reactivated the m2f wiki as READ-ONLY. This site has some good information for m2f 1.2, although some of it is outdated. It has been hammered a lot by spammers though, and maintainance was a pain, so I just turned it back on READ-ONLY.
• Added and polished several sections of the 2.0 development page. Be sure to check out the new m2f 2.0 design and the m2f 2.0 threading uses cases pages.
• Cleaned up and updated the announcements in the v1.2 and v2.0 development forums with download, install, etc. instructions (Note that v2.0 developers forum is accesible only for users in the developers group).
• Did other smaller housekeeping tasks.

We also configured (with m2f of course!) two new activity tracking forums:

• v2.0 SVN activity: It’s constantly updated with last-minute Subversion commits (Out-of-the-oven code)
• v2.0 Tracker activity: It automatically stores changes to our issue tracker, including bugs, new features, changes in our roadmap, etc.

If you wish to subscribe to those forums, just visit the Forum Subscriptions link on the forum (You have to be a registered user!).

Last but not least, if you want to keep yourself updated with latests posts in this blog, there is also a RSS Feed available. Until we build a wordpress channel for m2f 2.0, we’ll manually post on the forums a link here as well!

• Basic Mail to Forum using POP3,SMTP,Sendmail,mbox,phpBB2 and phpBB3
• Some basic channels like FileWriter (Export), Random Text Generator (Import)
• Basic RSS Feed import to any functional export channel (SMTP,phpBBx,FileWriter).
• Forum mirroring! Watch phpBB2 and phpBB3 talk together.
• Basic Admin Control Panel
• Message threading

For more details, check out our Roadmap in the tracker. For the PHP gurus out there, m2f boasts under the hood an impressive design framework. Just to mention a few tips:

• Connection to the web-based-channels like phpBB2 and phpBB3 is made using direct http connections! This means that there is no need to actually host those forums in the same server than m2f.
• Configuration of phpBB channels is EASY. Just upload one file, do a tiny include on one file, and you are done. (Note: phpBB2 install is a bit trickier because of some phpBB2 design flaws we must fix before connecting it to m2f) (Note2: In milestone 1 this is not as easy as it sounds, but milestone 2 will further simplify the task of configuring phpBB channels - Feature #137)

Of course, all of this is built upon some nice frameworks - read more about that on the Technology page. Last but not least an annoucement… I’m getting back at coding! Yeah, you’ll see some Pda0’s SVN checkins on our code repository activity forum

I built the Feed channel in a week with the help of George, with Unit and Acceptance tests and all, and it was lovely to see some rss feeds imported into my test phpBB3 forum!

Stay tuned!

.pd