WorxWare

Through this blog, I will be posting articles on technology, mainly related to email processing.

Please keep your replies related to the topic. This blog will not be used to provide any support or individual tutorials.

About five years ago, one of our clients contacted us about a project to minimize spam generated from their Feedback and Contact Us forms on their site. Not only were they getting more than 500 spam emails daily from the forms, but the forms were also being used to relay spam to large mailing lists showing the client as the sender.

This wasn't the first time we had been contracted to solve this type of problem.

It is rare to see a website without a form. Forms are the life-blood between your presence on the web and the prospects and clients that you want to interact with. Forms provide you with the data that makes or breaks your business. Forms can be used to collect user information. Examples are:

• Contact Us
• Feedback system
• Surveys

Forms can also be used to populate your databases. Examples are:

• Registration/Sign-up
• Data input of all types
• E-commerce

In the first set, collecting user information, the data collected is typically sent by email to the designated recipient, usually you or someone you appoint to get these types of info. emails. The forms require security to prevent hi-jacking of your form to deliver the hacker's email payload using your bandwidth and your identity.

In the second set, populating your database, the data collected is typically posted directly to the database. You may have a verification process where new accounts need to be approved before being live, but essentially the process is to take the user data typed in the form and post it directly to a database. These forms are particularly dangerous because a hacker can use the form to gain access to your entire database.

Back to the five year old project ... using a variety of techniques, we were able to eliminate almost 100% of all the hacks and spam emails. This hasn't been easy and we've changed strategies over time.

Our initial efforts were based on Javascript to both validate and format the data. That eliminated about 50% of all the emails. In analyzing why the Javascript was only 50% effective, we found that most hackers were already familiar with client-side validation. Using browser techniques, they launched the form, and then disabled Javascript to bypass the validation.

We then combined the Javascript validation (client-side validation) with server-side validation. Over a period of about 8 months refinining this technique, we were able to nearly eliminate 100% of the garbage emails.

That was four years ago. Since then we have changed our server-side validation several times using third-party scripts with some modifications from us. That was our proof of concept -- server-side validation works. To date, no one has been able to get past the server-side validation.

The first server-side validation script we used was the proof we needed that this technology is a must-have. Only one problem, though, is that third-party scripts typically are designed around the author's needs and not very flexible. Such was the case with the first server-side script we used. The validation rules were all based on some very rigid needs of the author, plus was tied to a templating system that forced a very rigid style of form design. For example, the templating system isn't able to handle multi-column forms, or radio buttons / check boxes horizontally. Plus, validation error messages were rigid in how they displayed.

Our second choice for server-side validation was newer technology and based on a more flexible templating system. It was closer to our needs -- but it lacked in validation rules and validation error messages were cryptic.

We have now totally eliminated client-side validation. The hacker's work-arounds for javascript based validation are too well known and client-side validation is now far far less effective rendering client-side validation as close to useless as possible.

We are also working on our our PHP-based server-side validation scripts that we expect to release as Open Source in 2011. If you have any feedback or features you would like to see, drop us a line.

Andy

Microsoft has recently made changes to their Hotmail ports and settings.

Our basic example on the PHPMailer website now is:

require_once('../class.phpmailer.php');
//include("class.smtp.php"); // optional, gets called from within class.phpmailer.php if not already loaded

$mail = new PHPMailer();

$body = file_get_contents('contents.html');
$body = eregi_replace("[\]",'',$body);

$mail->IsSMTP(); // telling the class to use SMTP
$mail->Host = "mail.yourdomain.com"; // SMTP server
$mail->SMTPDebug = 2; // enables SMTP debug information (for testing)
// 1 = errors and messages
// 2 = messages only
$mail->SMTPAuth = true; // enable SMTP authentication
$mail->SMTPSecure = "tls"; // sets the prefix to the servier
$mail->Host = "smtp.live.com"; // sets HOTMAIL as the SMTP server
$mail->Port = 25; // alternate is "26" - set the SMTP port for the HOTMAIL server
$mail->Username = "yourusername@hotmail.com"; // HOTMAIL username
$mail->Password = "yourpassword"; // HOTMAIL password

$mail->SetFrom('name@yourdomain.com', 'First Last');

$mail->AddReplyTo("name@yourdomain.com","First Last");

$mail->Subject = "PHPMailer Test Subject via smtp (Hotmail), basic";

$mail->AltBody = "To view the message, please use an HTML compatible email viewer!"; // optional, comment out and test

$mail->MsgHTML($body);

$address = "whoto@otherdomain.com";
$mail->AddAddress($address, "John Doe");

$mail->AddAttachment("images/phpmailer.gif"); // attachment
$mail->AddAttachment("images/phpmailer_mini.gif"); // attachment

if(!$mail->Send()) {
echo "Mailer Error: " . $mail->ErrorInfo;
} else {
echo "Message sent!";
}

Andy

As the developers, we obviously use PHPMailer extensively in our applications. In some, we were using HTML2Text, a very good utility available at chuggnut.com. For basic forms processing, HTML2Text is overkill and does not render the forms data properly -- particularly the tables, th, td, and tr tags.

We modified several functions that we use in our content management systems and in our own PHPMailer scripts and wish to discuss those here and provide them for your use: same license as PHPMailer, LGPL -- please attribute properly.

The two functions that we modified are:

1. A function that strips out the <body> tag through to the </body> tag, inclusive. We use this in a commercial email marketing application to strip out all the HTML tags above and including <body ... > and strip out all the HTML tags below and including </body> (meaning exclusive of the body tags) ... the modifications are to inverse the results returning only the inclusive portion.
2. A function that:
   • converts HTML entities to character representations
   • strips out all new line characters and spaces after the closing tag element
   • converts </td></tr> to new line characters
   • converts </td> to a colon and space
   • then strips all tags

The first function is to strip out certain code that is not processed by other HTML to Text conversion utilities. One example, is the <style></style> tags and everything contained within those two tags.

function _stripStartEndStr($str,$startTag='<style>',$endTag='</style>') {
/* Copyright Andy Prevost */
$startTag = strtolower($startTag);
$endTag = strtolower($endTag);
$lower_contents = strtolower($str);
// determine if a $startTag tag exists and process if necessary
do { $posStart = strpos($lower_contents,$startTag);
if ( $posStart !== false ) {
$posEndStart = strpos($lower_contents, $endTag);
$posEnd = $posEndStart + strlen($endTag) + 1;
$posEnd = $posEnd - $posStart;
// return stripped out tags and contents
$strPart = substr($str, $posStart, $posEnd);
$str = str_replace($strPart,'',$str);
}
} while (0);
return $str;
}

To use this function, derive your HTML the normal way, then convert it to text:

$html = {whatever you normally do};

$text = _stripStartEndStr($html);

The next function does the actual HTML to Text conversion. Note that it will render your tables reasonably, convert all HTML entities to characters (like &copy; to ©)

function _html2txt($html) {
/* Copyright Andy Prevost */
if (trim($html)=='') { return $html; }
$text = htmlspecialchars_decode($html);
$text = str_replace("</table>", "</TABLE>", $text);
do { if (strpos($text," </TABLE>")) { $text = str_replace(" </TABLE>", "</TABLE>", $text); } else { break; } } while (0);
do { if (strpos($text,">\n\n")) { $text = str_replace(">\n\n", ">\n", $text); } else { break; } } while (0);
$text = str_replace(">\n", ">", $text);
$text = str_replace("</tr>", "</TR>", $text);
$text = str_replace("</td>", "</TD>", $text);
$text = str_replace("</th>", "</TH>", $text);
$text = str_replace("</TD></TR>", "\n", $text);
$text = str_replace("</TH></TR>", "\n", $text);
$text = str_replace("</TD>", ": ", $text);
$text = str_replace("</TH>", ": ", $text);
$text = str_replace("</TR>", "\n", $text);
$text = strip_tags($text);
return $text;
}

... then add your HTML content, and add your Text content

$mail->MsgHTML($html);

$mail->AltBody = _html2txt($text);

Enjoy!
Andy

We are working on new documentation for all of our software. The documentation will include:

• Requirements
• Pre-installation instructions
• Installation instructions
• Accessing the installed software
• Basic usage and tutorials

The first set of documentation is nearly complete for our upcoming release of PHPMailer-ML version 1.8.

The documentation is being setup as a "knowledgebase" and you can preview it at:

http://www.worxware.com/kb/

[update: feb 04 2010] The documentation site software is not flexible enough to provide an organized view of the documentation the way we want it. We are working on another software platform at:

http://www.worxware.com/kbn/

Enjoy!
Andy