The m2f development team are delighted to announce the immediate availability of Alpha release 2.0.0a5. This is an alpha-quality build of the forthcoming v2.0 release, which will see the transformation of m2f from a simple phpBB -> email link into a multi-faceted message hub.

This release is aimed at alpha/beta-testers only. It is not ready for production use. Installation requires a long and precise list of instructions, and testing requires a number of dedicated resources, with potential risk to databases.

The release can be downloaded from Sourceforge at the following link:
https://sourceforge.net/projects/m2f/files/m2f/2.0.0a5/?

For an introduction to v2.0, help with installation, testing and setup, please see our documentation.

If you want to peek at the development plan, please see the roadmap on the m2f Issue Tracker.

Enjoy!

m2f 2.0.0a5 (r878) Release notes

Milestone 2 Alpha release:
* Fixed bugs 190, 187, 126, 54, 53, 42 and others non-documented issues
* Filter initial support
* Attachment support (Tested on phpBB3)
* Various refactorings
* Feed import channel (Export is buggy)
* Some UI improvements to the admin CP
* Admin CP authentication
* New GPL License
* Security: Publicly-accesible scripts moved to /public (Including admin cp access)
* Compatibility: Tested with latest php, phpBB3, mysql (Under Windows)

I worked a little on the issues reported (http://tracker.mail2forum.com/projects/m2f/issues) as well, but there’s yet much to analyze.

I updated the roadmap, stripping a lot of functionality that probably will be needed but falls outside the bare-bones but usable alpha I want to release to the public ASAP. Check it out: http://tracker.mail2forum.com/projects/m2f/roadmap and comment in the forum. It still needs some work, but I think it’s a good start.

In all, alpha “4.1″ should be ready for deployment. I’ll wrap it up and publish it in a few days, then start working on the rest of the features of the roadmap.

.pd

A lot of time has passed, and there is a considerable amount of work in the setup process of the development environment. I will have to check the libraries and php version conflicts, as well as phpBB3 latest changes. After that, update the roadmap and issue tracker so that it reflects our new goal: To get a bare-bones phpBB3< ->email solution ASAP.

If I find some time, I would like to do some housekeeping on the blog as well to make things easier for newcomers or long time comebacks – same thing in the forum.

I also might open up the alpha testing to the public and see if that works out. In the past we tried to stay away from that as sometimes an over-enthusiastic but incompetent tester can give the dev team more headaches than solutions but hopefully some m2f veterans will come in our aid.

Do take into consideration that I haven’t coded a line for several years, and thus if you feel now it is the time to step-in and help us out, please don’t hesitate and post in the forum.

I know there some power users out there that are already using phpBB3 with m2f 2.0. I would appreciate you drop by the forum and tell us your experiences so we can get the most out of our development efforts.

In order to try to support a more sustained development in the future, I’m also analyzing the licensing scheme in order to get some revenue that would allow us to buy dev-time from php specialists and thus accelerate the whole process. Although I would love to maintain m2f 100% free, this might not be feasible considering our development history and if we want to get a final version at all. As in the rest of the fronts, I’ll keep you posted.

Cheers, comments, rants etc. would be appreciated! Tell us what do you think in the forum.

.pd

Nevertheless, I will try to move the development forward a bit, albeit not with the rigurosity and elegance of George’s software design. Code like hell activated folks! I built a “pda_mess” branch where I’ll start my work.

The goal: To have phpBB3< ->mail working asap. Hopefully later on George will be able to clean up the mess and integrate the development into the main “trunk” branch.

Yesterday I updated some of the code to support php 5.3.0. Also updated Zend framework. It seems that php is becoming increasingly strict with semantics, also there are several functions that are becoming deprecated. split() is giving me a headache right now, it’s beign replaced in favor of preg_split() is more powerful and faster. Sadly, split() is being used in the swift email library and in simpletest as well. All right, time to update the rest of the libraries…

.pd

As you already know, in these open source projects sometimes real life gets in the way, but after almost 9 years of development we can say with confidence that we always return to m2f!

Well, m2f 2.0 is a complex design-focused message hub. Email handling complexity by itself is a big deal, and that is only one (albeit one of the most important) of the channels that m2f 2.0 supports. So, prioritizing is an important part of our development plans, remember we have only one senior developer (George) and one junior developer (me) right now. If you need m2f and can give us a hand (ie. 2-5 hours a week) then be sure to drop us a line – pm GeorgeC or Pda0 on the forum.

The current state of m2f 2.0 already allows for basic chaining of email, phpbb2, phpbb3, feed and other minor channels. Messages get threaded and attachments get processed, in a basic level (For more info read about MIME handling on previous blog posts). In general, the core m2f functionality is there, even though still in a immature state. One of the fun things that you can already do, and wasn’t possible with m2f 1.x, is binding phpbb2 and phpbb3 together (Or two phpbb3 installations together).

So, what’s next. There so many tasks lurking on the tracker, that we need some strategy to slowly crunch each of them. At the moment we have three objectives that drives our development priorities:

1. The requirement of one of our sponsors that needs m2f working with mailman+phpbb3 asap.
2. Build a very simple m2f beta-quality solution involving only some of the functionality.
3. Reorganize milestones from functionality classes to integral and incremental functionality releases.

Point 1 mainly focuses in mailman<->phpbb3 integration.

In point 2, we are building a feed->phpbb3 solution that can be released as a separate, beta-quality MOD release. Even though this task will divert us a bit from the main line of development, it will help us get a bigger userbase of users/testers and also prove the new concepts in production environments.

Point 3 deserves some explanation. If you look at the milestones in the tracker, you will see that requirements are grouped thinking in the types of functionality that are needed. For example “Email features” or “messagers & filters”, and so on. This helped the development a bit, as developers can focuse in one type of problem at a time, but it also greatly reduced the integral functionality of a m2f release. We want to change that strategy so that every m2f release has a bit of functionality from each class, and thus is more practical for normal use.

Other than these, we’ve been working in finishing Feed channel and other general design improvements. In latest SVN, Feed no longer (re)imports feed posts that were already imported in previous chain runs. Also there are big improvements in the html->bbcode parser.

Stay tuned!

.pd

As I’ve been doing a huge amount of work on the email processing feature of m2f 2.0, I’d like some assistance with testing.

The new functionality is in the Email Import channel. In short, it will convert a multipart email (i.e. one in plain text as well as HTML, or an email with attachments) into a single, flat message suitable for using later in an m2f chain.

For the present, all attachments are stripped from the incoming message, and saved into a directory on your filesystem. There’s a new File Manager class which servers up the files, given a unique URL. For example, on my test server, an attachment will be given a URL like this:

http://m2f/download?file=5863b365ee66b7a28546b4f07e09913b.

If that URL is entered into a browser window, m2f will offer the file for download.

I’d like you to test the following functionality:

• check that all processed emails are legible, and well-formatted
• check that all attachments are correctly stripped, and are downloadable from the specified URL

Testing the MIME handling

There is a specific sequence of operations you must follow to properly test the new functionality. Please follow the instructions carefully. You will need an archive of old emails to test with, in mbox format. Please try to test at least some with HTML parts, and attachments, where possible. This is the main focus of the test!

1. Work with a copy of your mbox file, as we will be deleting messages individually from it as the test progresses.
2. First, it’s imperative that you do a new checkout from SVN (the MIME parsing feature isn’t yet available in a release), and run the entire test suite. Any errors must be reported and resolved before there’s any point in proceeding.
3. Set up the following chain using the m2f Admin Control Panel:
   • Import channel: M2f_Module_Channel_Email_Import_Mbox
     • set filePath to the path to your mbox file
     • set maxMsgs to 1
     • set deleteMsgs to false (i.e. do not check the box)
   • Export channel: M2f_Module_Channel_FileWriter
     • set the filePath to the path of a file which is accessible using your webserver, with the suffix “.html”. For my tests, I use a file which can be accessed using the URL http://localhost/mime_test.html.
4. Run the chain. This should result in the contents of the first email in the mbox being parsed by m2f, and written to the file specified in the Filewriter export.
5. View the HTML file in your browser. If the original email contained any HTML, the generated file should be in HTML as well, and should already look good in your browser. If the original was plain text only, the contents of the HTML file will actually be plain text, so you’ll need to View Source in your browser to see the correct output. Please ignore the fact that a plain text message looks crap when viewed as HTML – that’s expected behaviour!
6. Check that the message looks good – it should be legible, with no missing images or text. Please note that it will not necessarily look the same as in your email client, as at present I’m stripping all CSS from the HTML headers. This may change in future.
7. Check that any attachments were correctly scrubbed, and that they are now accessible using the given URL. A block of text like this:
   An attachment was scrubbed:
Name: my_pic.jpg
Type: image/jpeg
URL : http://m2f/download?file=98070067013836d34358de6c092ef3c0should have been added to the bottom of the message for each attachment. Copy the URL, paste it into your browser, and you should get the option to download the file. Check the file is of the correct type (i.e. Microsoft Word documents open OK in Word, etc.), and that the contents is as expected.
8. If anything went wrong with the parsing – if the email looks weird, images are missing, attachment are not accessible, etc. – please do the following:
   • copy the RAW source of the email in question from your mbox file. It will be the  text starting from line 1, and going up to the next instance of this string: a new line, followed by the word “From”, followed by a space.
   • paste the raw email into a new text file. Please use the same encoding as the mbox file – if in doubt, use UTF-8.
   • replace any sensitive data with dummy data – but please don’t in any way alter the syntax of the email
   • email the file as an attachment to me at george@mail2forum.com, explaining what the problems were.
9. If the output looks OK, delete the raw email from the mbox (see above for how to identify a single email). This should leave the first line of the mbox file as the “From …” line of email number 2.
10. Run the chain again for the second email.

I’m very grateful for your help with testing this complex bit of code!

After the initial importing of a message, m2f analyzes the message content in order to understand its structure. If the message is being imported from an email channel, that means looking the current format of the message (ie. html, plain text and so on), checking if the email has attachments (normal attachments or inline attachments), etc. The way that emails handle multiple formats and attachments is called MIME Handling. As quoted from wikipedia:

Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of e-mail to support:

• Text in character sets other than ASCII
• Non-text attachments
• Message bodies with multiple parts
• Header information in non-ASCII character sets

Yes… daily life email messages are more complex than you could think! A typical email could have several parts, including a plain-text AND html-version of the message, inline images, file attachments and so on. m2f goes through all the parts and cleverly guesses which parts should be used for the message conversion process, which should be stored for later use, and which we’ll actually discard as message scrap.

m2f is being designed to alter the original message as well, which is an important matter for security’s and flexibility’s sake. For example, it will be able to sanitize html messages in order to try and filter out common code-inyection vulnerabilities (such as Cross Site Scripting). This will be configurable in the future.

Right now m2f mainly passes html as-is to the export channel, but first it strips all attachments and stores them in the m2f attachments directory. It then adds a link to every attachment at the bottom of the message, that points to the attachment that was stripped. This might be early attachment management, but it is working very well! Check this out:

Original Outlook Email

m2f Processed Email

Did you notice the “scrubbed attachment” on the Processed Email? Of course this is bare-bones implementation, there is still a lot to do, but the most difficult part (MIME Handling!) is almost over. The screenshots above are a result of a “mbox->filewriter” chain configured with current SVN

.pd

These have been busy days! George is finishing off MIME handling of email messages and building the general architecture that will eventually support full attachment management between channels.

I have been working in the Feed class, adding export functionality and also did some heavy refactoring to get cleaner code. There are still things to be done, in particular the configuration of the channel in the Admin-Cp, but we are almost there.

I would like to tell you a story about what’s involved in building a channel with the current state of the code (alpha 4):

• First of all, as always, you have to draw a basic sketch of what you want to do. For Feed I knew I wanted to import web feeds (mainly rss) into a forum, and export, say, latest 10 posts in a given forum out to a rss xml file.
• It is important to think whether your channel will only import, export, or both!
• Then, you build the framework of your channel, classes and interfaces, in the “lib\M2f\Module\Channel” directory. The tests will go into “\tests\lib\M2f\Module\Channel” directory.
• Then you can start building up the tests, then adding code to make them pass. You always build the test before writing actual code because it cleans up the design and also helps developing very robust code!

m2f channels revolve around objects called “Generic Messages”, which are, in essence, a data structure that holds all the information a message could eventually have. The channel converts messages from other sources such as Forums and Email into a Generic Message, or a Generic Message into a post or email message.

Having a Generic Message structure is the central approach of how m2f bind channels together. Instead of building a n-to-n format converter between channels, we decided using an intermediate format that every channel must know about.

So, for import channels, the difficulty resides in connecting to the remote software (via database link, POP, propietary API… you name it) and extracting messages in the original format. Then, convert that format into a Generic Message that can be further processed.

For export channels, the algorithm must take a Generic Message, convert it to the destination format, and finally connect to the remote software to submit the new message.

This might sound easy enough, but there are several pitfalls, including: Message threading, ugly code at the remote software side, attachment handling, weird uses cases (what happens if….?) and others. So, you have to take these into consideration if you want to develop a robust channel.

.pd

Installation & Basic Chain

The video shows the basic installation procedure, and setting up a test chain with a Random Text Generator input and a File Writer output.

www.youtube.com/watch?v=mjk-d5cO2q0

m2f 2.0 Alpha 4: mbox email to phpBB 3

An introduction to m2f v2.0 alpha 4, released Feb 11th, 2009. The video shows the connection procedure with a remote host such as phpBB, and setting up a chain from a mbox email file to a phpBB 3 forum.

www.youtube.com/watch?v=odkcA-GIZu4

The m2f development team are delighted to announce the immediate availability of Alpha release 2.0.0a4. This is an alpha-quality build of the forthcoming v2.0 release, which will see the transformation of m2f from a simple phpBB -> email link into a multi-faceted message hub.

This release is aimed at alpha/beta-testers only. It is absolutely not ready for production use. Installation requires a long and precise list of instructions, and testing requires a number of dedicated resources, with potential risk to databases. Support will not be offered unless accompanied by bug reports on our Issue Tracker. Please only download this if you are a member of the Beta Testers group.

For an indication of the new features in this release, please see the roadmap on the m2f Issue Tracker. For an introduction to v2.0, help with installation, testing and setup, please see our documentation.

The release can be downloaded from Sourceforge at the following link:
https://sourceforge.net/project/showfiles.php?group_id=68964&package_id=67748&release_id=660489

If there are any PHP developers who would like to be involved in the development of m2f v2.0, please get in touch with Pda0 or Georgec on the forum!

Enjoy!