Friday, September 6, 2013

Dada Mail, v6.6.0 Released!

I've just put out a new release of Dada Mail - v6.6.0. Its  a web-based mailing list manager, written in Perl. You can check out/fork/push/pull the source code here. Some of its main goals is to be pretty easy to install by mere mortals, easy to work with, yet pretty darn powerful and flexible, to fill a ton of different needs - it's not really trying to be a replacement to the creaky Majordomo, or Mailman. 

I'm heading up to my 14th year of developing the application. It's currently free to download from its own support site as well as from various installer services that have popped up in the last few years (Mojo Marketplace, Installatron, Softaculous, etc) Follow along with the project on Twitter, or hey - we have a mailing list too!

There's lots of things a like about working on it, but the application has become quite monolithic, and a little less flexible in its underlying code, than I'd like. In the next few months, I'm hoping to alleviate that, without falling into the problem of totally rewriting the app (it would be almost impossible to do so, with a one-dude team).

My initial idea is to move from basically no framework, to working with CGI::Application. That should help me gain some sort of framework, and a way to break the overgrowth of parts of this app, into more manageable pieces. It'll also hopefully give me a bridge to be able to have Dada Mail run as a CGI script (how people are used to running Dada Mail), as well as a PSGI app (a great alternative, that has a TON of benefits, one being SPEED!).

From there, we'll see how it works, but it may be time to move on to some more whiz-bang things that a framework like Mojolicious would provide. The bummer with trying to move to Mojolicious is that the Perl version required is rarely met on the shared hosting accounts that the majority of people who run Dada Mail are on. Saying, "hey, just install perlbrew!" is a far cry on what lots of people are capable of. They're used to a one-click install. Sigh.

Maybe cPanel, being a Perl-based shop, knows of a better way to have available a more up-to-date Perl version? A how-to, so not to fudge this up, would be super-helpful and somewhat break huge shackles from those that ship apps for this types of hosting situations. (maybe you know?)

Without further ado, here's the changelog from this last release:



Summary 6.6.0

This release of Dada Mail focuses on three main features:

Features

Mass Mailing Enhancements: Sending Error Detection and Reporting

Dada Mail has enhanced its sending error detection during mass mailings. Due to how many messages go out during a mass mailing and how fast you may be sending out these messages, problems with mail sending could potentially happen. Historically, Dada Mail has been less than bright than it could be when handling and reporting these problems.
Dada Mail is now tuned to catch addresses causing problems and perform some intelligent tasks:

Problems?! Try again.

If a message to an address is not sending out correctly, Dada Mail will now wait a small period of time (a few seconds), and try sending twice more. This should help with any very temporary problems dealing with your mail system itself (your own MTA, an outside SMTP server, or service like Amazon SES). Hey, things happen, every once in awhile. The idea is that brief hiccups on your end, shouldn't mean a subscriber misses receiving an important message from you.

Still having problems? Give it a rest and try again, one more time

If, after three tries sending the message, the message still doesn't get sent, Dada Mail will now temporarily stop it's entire mass mailing process and start again, after a slightly longer amount of time (a few minutes). This is to make sure there isn't a problem with Dada Mail itself - the resources its taking on the server, or any other very strange, very fringe problems.
After this wait, Dada Mail will start its mass mailing process at the address that was having the delivery problems and try for another round of three more times - just to make sure. If this is successful, great! Business as usual. If the address fails in sending, the address is logged, and skipped over and the next address is then sent to.

Sending Error? Log and Report.

This is a marked improvement over what Dada Mail has done in the past: basically do nothing very intelligent when an address fails during sending. Addresses that fail are now reported, using the Tracker plugin, which allows you to see exactly which addresses are causing problems (this data can be exported as a .csv file, as well!) and what percentage of your total mailing list is having problems with sending errors.
Now that you know this information, you can then take some action to fix it. The majority of problems on cheap, shared hosting platforms are related to mass mailing speed: you're sending too fast.
This is also the eaisest types of problem sto fix, as Dada Mail supports changing how fast a mass mailing goes out, through it's batch sending system. For more complicated problems, you can then look in Dada Mail's error logs for more details - Dada Mail's Log Viewer plugin also makes this easy.

Track it.

Since sending errors are now reported, the reports created by the Tracker plugin can now be more accurate. Message opens, bounce rates, etc. percentages now do not count addresses that failed to even be sent out by Dada Mail, giving you a better idea on how many messages were interacted with, compared to how many were sent out, correctly (rather than the full total).

Feedback Welcome!

We're sure we'll be getting a lot of feedback from users that are just noticing that they may be having problems on their end with sending. We're hoping that armed with this new data, that deliverability will increase.

Tracker plugin enhancements: Sending Errors

Along with logged Sending Errors, the Tracker plugin has been updated to report those errors in a easy-to-digest way:

"The Basics" Pie Chart

Sending Errors are first shown reported, per message, in the first Pie Chart at the top of the screen. The pie chart is composed of Delivered Messages, Hand/Soft Bounces, as well as addresses that had Sending Errors.
Sending Errors are also reported in the right-hand table, under, Sending Errors as both the total number of Sending Errors, and the percentage compared to how many addresses were sent to.

"Sending Errors" tab

Sending Errors also has its own tab, underneith, The Basics. This tab will both list the addresses where Sending Errors happened, as well as a pie chart, broken down by domain. There's also an option to export these addresses as a .csv file.

Other Tracker changes: more accurate report percentages

The Tracker plugin now shows percentages to the nearest tenth of a percent, rather than rounding up numbers to the nearest whole number. This helps in getting a better picture of what's happening with your mailing list: things like Sender Errors, Bounces and Unsubscribes could be a very small amount of activity, when compared to how large your mailing list is.

Working with Subscribers: More Powerful Tools

List Owners now have much more powerful tools to work with individual subscribers of their mailing lists, as well as the ability to work across different mailing lists. The completely rewritten Membership screen now has four separate tabs to work with your mailing list member: Membership, Subscriber History, Mailing List Activity, Profile.

Membership

The Membership tab allows you to work with the email address of one of your mailing list members. Here, you may Add, Update or Remove the address for your mailing list. A Dada Mail mailing list is actually composed of many different sublist types. So, for example, when you send a mass mailing out, you're sending a message out to your Subscribers sublist.
But, Dada Mail has other sublists that provide different roles. For example, Dada Mail's Black List allows you to have a list of addresses that aren't allowed to join your list. This is also known as a, Suppression List
The, Authorized Senders sublist is a list of addresses that are allowed to send out a message from their own mail reader, without having to log into Dada Mail, that then gets delivered to your Subscribers (this is done with the Bridge plugin, which is bundled with Dada Mail)
The Membership tab allows you to Add/Update/Remove addresses across these sublists. If you're logged into a mailing list with the Dada Mail Root Password, you'll also be able to do all things things, across all your mailing lists.
Dada Mail is smart when allowing you to work with the addresses that make up your mailing list, as these addresses are the most valuable asset of the whole system. Checks are done to make sure you're not going to mess up subscriptions, and all these functions only happen after Dada Mail validates your request and you confirm the changes.
Listed in the Membership tab will be all the sublists an individual email address is a member of, for the Mailing List you're currently logged into. Be aware that one address may be a member of multiple sublists.
For example, an address may be both a Subscriber (Subscribers receive mass mailings sent to the mailing list), an Authorized Sender (Authorized Senders are able to send messages via Bridge, as well as moderate messages sent via Bridge to a mailing list set up as a discussion list), as well as being on the White List (Only White Listed members are allowed to subscribe to a mailing list).
An address could also be simply on the Black List, with no permission to be a member of your Subscribers, until they're first removed from the Black List and added to the Subscribers sublist.
If an address is currently on the Subscription Requests sublist, they will be awaiting approval from the List Owner to join the Subscribers sublist. You may Approve or Deny this request on this screen as well.

Subscriber History

Dada Mail logs the subscriber history of an address: when the address was added/updated/removed from one of your mailing lists and their underlying sublists. This, "paper trail" helps out when painting a picture of how this address has become a part of your mailing list. Information recorded includes the time of the add/update/removal, IP address of the request, and what exactly was done with the address.
This information can also be exported in a .csv format, to be opened and worked with, in any application that understand this format - things like Excel or other spreadsheet applications.
If you have logged into your mailing list with the Dada Mail Root Password, you may view the history for the mailing list you're currently logged into, as well as all mailing lists - the latter can also be exported to .csv format.

Mailing List Activity

The, Mailing List Activity tab shows you how the subscriber has interacted with the mass mailings you've sent. Things like, Opens, Clickthroughs, Unsubscriptions, Sending errors, Hard/Soft Bounces are tracked for each message you send out. This information is shown for each recent mass mailing that's sent.
Like the Subscriber History, this data can also be exported in .csv format, to be opened up in an outside application. Data that's exported includes the data and time of the activity, what type of activity this was, if it was a clickthrough, what URL was clicked through, the message id of the message, as well as the IP address the subscriber was coming from.

Profile

The Profile tab allows you to work with the Subscriber's Profile. Dada Mail's Profile system allows you to save additional information, other than an email address, that's association with your subscriber. These fields are pretty free-form, so you can add whatever you'd like: First Name, Last Name, City, State, Favorite Color, etc.
These fields can be edited by the user themselves, as well as in this tab. You may also change the password used by your user to access their own information.

RESTful Subscription Interface EXPERIMENTAL

Dada Mail has a new EXPERIMENTAL RESTful interface to its mailing list subscription system, which should allow you to use it as a service in another app of yours. For more information, see the Subscription Cookbook docs:
http://dadamailproject.com/d/COOKBOOK-subscriptions.pod.html
Examples of how to use this API are included, written in JavaScript and Perl. A fully-functioning demo, complete with all the trimmings is also included in this release, that you may use to explore the service.
Earlier REST and SOAP experimental client/server scripts, located in the dada/extras/scripts/subscribe have now been removed. For this new RESTful interface, no additional scripts are needed to be installed.

Administration Screen changes

Changes have been made to the, Administration screen - if you are already logged into a mailing list, you'll be redirected to the default list control panel screen (usually, Send a Message), rather than to the Administration screen, that simply tells you you're logged in and provides a link to go this same screen.

Black/White List Definition Changes

The Black List can currently hold email addresses,
        user@example.com
or parts of email addresses, like this:
        user@
        @example.com
It wasn't entirely clear what was meant by, "part" of an address, so this has been clarified:
The part of an email address must include the, @ symbol, to remove the ambiguity of if you want to Black List the user or domain part of the address and something like this doesn't happen:
        example.com@someotherdomain.com
So the following are now not valid Black List (or White List) entries:
        user
        example.com
If you do have entries like this in your current Black List/White List, they simply won't do much of anything.

Installer

Enable Debugging Options

The included Dada Mail Installer now allows you to enabled Debugging Options for Dada Mail. These debugging options add extra log information in Dada Mail's own error log, and includes both modules that make up Dada Mail, as well as bundled CPAN Perl modules. This option is located under the, Advanced Configuration (Optional) heading, and is available in our Pro Dada distribution.

JSON module installed if needed

The JSON CPAN modules are used heavily in Dada Mail, but is not installed by default on many hosting platforms. To make things easier, the Dada Mail Installer will look to see if they're already installed system-wide and if they're not, it will move a bundled Pure-Perl version of the modules into its own perllib, thus making it available to be used. This saves some time from having to manually doing this same thing yourself, every time an upgrade is done.
It's still suggested to install the JSON Perl module yourself to get the XS-version for speed, either manually, or using cpan/cpanmin

Changes 6.6.0

The Dada Mail Manual

The Dada Mail Manual has been updated to reflect the major overhaul fo the Membership list control panel screen.
The, Using Tracker chapter has also been updated to reflect the new features in this released (logging and reporting Sending Errors)
The, Mailing Sending - Options screen has also been updated to reflect the current options available.

Profile Fields

In previous versions of Dada Mail, the app had a curious, if not downright broken behavior when it come to Profile Fields:
If a subscriber was a member of one mailing list, and was then subscribed to another mailing list, any saved Profile Fields tied to that address would be lost - even if that new subscription didn't have any new Profile Fields passed to Dada Mail.
This behavior has now changed:
If an address, that has saved Profile Field information, and is subscribed to a mailing list, that original information will now be preserved. If an address is subscribed, but had no profile field information saved, any new information would then be saved.
It's unclear yet what to do with a subscriber, that has saved Profile Field information that's subscribed to another mailing list, along with new Profile Field information - should the new information be used instead? Should the old information be kept? Shoudl some sort of sophisticated folding of data happen? Should it be a pref. somewhere? (per mass import, in the list control panel?)
We're not sure yet, but we encourage feedback.

uncompress_dada.cgi script

The helper script named, uncompress_dada.cgi is kept intentionally simple, so as to be super-quick to download, and easy to read along, if little niggling changes need to be made for your environment. We've changed a few things to make things more flexible:
Some people moving from Basic Dada Mail, to Pro Dada mail noticed that if BOTH the dada-6_x_y.tar.gz distro and pro_dada-6_x_y distro are present in your installing directory, the dada-6_x_y.tar.gz will be used, causing confusion. uncompress_dada.cgi will now explicitly look for the pro_dada-6_x_y first.

Bugfixes 6.6.0

"Your Request For Subscription is Complete" HTML screen completely missing

https://github.com/justingit/dada-mail/issues/398

""Welcome" Message to Addresses Subscribed in the List Control Panel" does not provide unsub link, warning in the list control panel

https://github.com/justingit/dada-mail/issues/395

Mass Mailings sometime Reload, without end - browser says, "Reloading Mailing..."

https://github.com/justingit/dada-mail/issues/399



6 comments:

Unknown said...

Hi

CGI::Application is not supported.

I re-wrote it as CGI::Snapp, and CGI::Snapp::Dispatch.

There are also a set of demos:
CGI::Snapp::Demo::*.

I only ever use it with Plack.

I've been tempted several times to switch to Mojolocious, but am not yet convinced it offers quite enough benefits, despite how impressive it is.

Unknown said...

Blogger displays an error (something like):

Memcache entry is null for Form Restoration.

when I tried to log in to Google, so we'll see if you actually see my comments.

This page has my original comment on the left, saying it was saved, so perhaps it did work.

The Perl Hacker Painter said...

Cool, Ron, I'll check it out!

The Perl Hacker Painter said...

Ron, there may be two things that will stop me from using CGI::Snapp:

one is the XS requirement - that'll kill me, as Dada Mail is used on more shared servers without access to a C compiler (or users that know how to use one)

The second is the absence of plugin support.

The last one may not be a big deal breaker, but one of the problems I'm hoping to solve when moving to some sort of framework is to break up the enormous monolithic main cgi script into manageable chunks - having a 12,000 line cgi script is embarrssing! :)

I thought that CGI::app had a pluing to allow you to keep separate run modes in separate files and then lazy-load them in, but I can't find that plugin on CPAN, anymore.

dolmen said...

Sympa is an alternate strong web based mailing list manager in Perl.

The Perl Hacker Painter said...

Ouais! Je sais de l'app, "Sympa".

Sympa!

Mais, C'nest pas la suject ici.