Introducing YippieMove '09. Easy email transfers. Now open for all destinations.

I think it’s fair to say that few people outside of Google have more experience of working with Google’s IMAP implementation (GIMAP) than we have. Since we launched YippieMove more than six months ago, we’ve performed a lot of transfers to Google using IMAP. Truth be told, we’ve spent a lot of time trying to find workarounds for bugs in Gmail IMAP implementation. In this brief blog-post, we will explore two bugs that we’ve reported to Google, but which Google seems to have little interest in fixing.

Inconsistency between SELECT and CREATE

The first bug might not be very juicy but it took us a while to recognize it. What we first thought was a bug in our system turned out to be a bug in GIMAP. As it turns out, in GIMAP, SELECT is case sensitive, while CREATE is not. Here’s a brief example to illustrate the bug:


0001 SELECT "INBOX/Sales Invoices"
0001 NO Unknown Mailbox: INBOX/Sales Invoices (Failure)
0002 CREATE "INBOX/Sales Invoices"
0002 NO Folder name conflicts with existing folder name. (Failure)

In this case, there is already a folder named ‘INBOX/Sales invoices’, but since SELECT is case sensitive, and CREATE is not, we were unable to select the folder and with an upper case ‘I’ and at the same time unable to create the folder.

We first encountered this bug when a user migrated from a case sensitive IMAP server to Gmail. Hence we could have two folders named ‘foo’ and ‘Foo’ without it being any problem until we tried to copy those to Gmail.

Rejection of random messages

The next bug is of far more serious nature and it is a bug that we run into everyday. We’ve spent a significant amount of time trying to narrow down this problem, but almost entirely without luck. What happens is that GIMAP decides to reject certain emails. Sometimes the upload works when retrying later, sometimes not. To make this even more interesting, GIMAP can reject emails that it just gave us. For instance, let’s say we’re copying messages from [email protected] to [email protected]. Even that a GIMAP just gave us a particular email, another GIMAP server rejects the same email. Strange, isn’t it?

When I said that we’ve spent a serious amount of time trying to narrow down the problem, I was not kidding. We’ve done statistical analysis on tens of thousands of messages trying to find some kind of pattern (including trying to find a correlation between the content in the header and the body), and still no luck.

That said, the failure rate is quite low. Out of all the messages we upload, only a small fraction gets rejected (a quick database query reveals that it’s currently at 0.04%). And if a message gets rejected, we clearly state what message we failed to upload in the Transfer Report that we send out to our customers upon completion.

In Google’s defense, they do state that they do not officially support ‘upload of messages,’ but that is a quite weak argument, as pretty much any other IMAP-enabled service on the market supports this. Not to mention that without APPEND, simple drag and drop operations may fail in email programs.

For the curious geeks reading this article, our software does comply with RFC 3501, as well as other related RFCs.

As a side note, we’re not the only ones experiencing the APPEND bug. The Google Group discussion for Gmail IMAP and POP is full with threads regarding this. The only official-looking response given multiple times by wár17 § is to upload in small batches as you otherwise is likely to hit the bandwidth limit. However, we can testify that this is not the case as the number of messages and the size of the mailbox seem to have little effect. We’ve had this problem with small mailboxes (<10 messages and only a few hundred bytes) and at the same time had mailboxes that exceed several Gigabytes running through flawlessly.

Update: While it seems like the rejection of emails is somewhat random, there are certain emails that are more likely to get rejected. The other day we ran a transfer of a folder including 600-something bounced messages. They were all rejected.

Author: Tags: , , ,
Introducing YippieMove '09. Easy email transfers. Now open for all destinations.

What’s better than having a job that you love? How about not having to work, and still make money? We want to give you this opportunity. Ok, we might not be able to make you a millionaire, but if you are successful, you will be able to make decent money while literally doing nothing.

The way you would be able to do this is though our brand new affiliate program. It’s dead simple. You link to us on your website using your own custom link that we will provide you with. Then, every time a customer comes to us from your site and we make a sale, we will give you 10% of the sale. Are we great or what? Other affiliates programs, such as Amazon’s only gives you 4%.

Affiliate status page

The Affiliate Program status page.

Convinced yet? If so, just head over and sign up right away.

Author: Tags: ,
Introducing YippieMove '09. Easy email transfers. Now open for all destinations.
Jan
13.
Comments Off
Comments
Category: Uncategorized

Shortly after getting iReport running using the fix described in this article, we ran into a new problem — the Postgresql support. After some Googeling, it became clear that one had to add the the driver for Postgresql by hand from here. If not, you will encounter this error message:

Error: java.lang.ClassNotFoundException: org.postgresql.Driver

When adding the driver, beware of what version you’re selecting. After banging our heads in the wall for a while, we realized that we picked the wrong version. On Mac OS X 10.5.6 (with all latest updates installed) the version you’re looking for is the ‘JDBC3′ branch (assuming you’re using the java version that comes with OS X). Once we figured this out, the installation was easy. First, download this file. Then you need to move it into the ‘lib’ folder in iReport.

If you’re a console user like us, here are the steps:

cd iReport-3.0.0/lib
wget http://jdbc.postgresql.org/download/postgresql-8.3-604.jdbc3.jar

Easy as pie.

Author: Tags: , , ,
Introducing YippieMove '09. Easy email transfers. Now open for all destinations.
Jan
12.
Comments Off
Comments
Category: Uncategorized

If you’ve never heard of iReport and Jasper before, you really ought to take a look at it. It’s a really impressive suite of reporting tools that can generate reports from pretty much any data source out there.

Almost exactly a year ago, we wrote a similar article on how to fix the launcher in version 2.0.x. While I’m sure Jasper improved iReport a lot during this time, they also managed to break the launcher in new ways with version 3.0.0.

This is how you can get it working (assuming you’ve downloaded it):

tar xvfz iReport-3.0.0.tar.gz
cd iReport-3.0.0/bin
awk ‘{ sub(“r$”, “”); print }’ startup.sh > startup2.sh
chmod +x startup2.sh
./startup2.sh

The commands above fixes two problems with the launcher. First we convert the line feed from DOS format to UNIX format (the awk-part). The second problem was that the launcher was not executable. If you just make the original launcher executable (chmod +x startup.sh), you will end up with this:

-bash: ./startup.sh: /bin/sh^M: bad interpreter: No such file or directory

Good luck, and have fun generating all those new cool reports.

Author: Tags: , ,
Introducing YippieMove '09. Easy email transfers. Now open for all destinations.

If you’re a regular here at Playing With Wire, you’ve probably already read our articles about Cacti. While Cacti does do a great job on visualizing load on your servers, it does not provide (by default) alerts when a server goes down.

When we launched YippieMove we quickly realized that we needed a reliable 3rd party that could ping our servers from several locations across the globe to ensure that we were not experiencing any problems with the access to our site. As we are quite tech-savvy here at WireLoad, we had a hard time justifying paying more than a few bucks per months for a service like this, since the service is so easy to write (we actually did write our own uptime-monitor with alerts a few years back using Curl, Crontab and some other tools, but would rather outsource this service).

So the search began. We required a few thing for this service:

  • Several servers across the globe that ping our servers.
  • Cheap. Preferably free (we don’t mind some ads).
  • Decent statistics showing response-times etc.
  • Reliable alert system by e-mail (luckily most US Cell providers allow you to send email to your phone, using [email protected].)
  • Must allow monitoring of both SSL and non-SSL servers.
  • A minimum of 4 monitors (we needed to monitor playingwithwire.com, wireload.net, yippiemove.com [with and without SSL]), but it would also be great if we could monitor our mail-server.
  • The more frequent the pings the better.
  • No back-links required.

One of the most impressive sites we found was Pingdom, a small Swedish firm that is trusted by companies such as IBM, Loopt and Twitter (wow, they must spend more bandwidth on alerts than pings with Twitter for sure). What we really liked about Pingdom was the general look and feel of their site. It feels fresh, responsive and reliable. The pricing is definitely within reason: they charge $9.95 for their Basic plan, which includes 5 checks and 20 SMS.

The next site we stumbled upon was SiteUptime. The site has a decent look and feel (but does not come close to Pingdom). After examining their pricing, we realized that we needed their Advanced plan, since none of their lower plans allowed SSL monitoring. The price for this plan is $10 per month. While their site and visualization does not come close to Pingdom, they do give you 10 monitors, as apposed to 5 monitors with Pingdom, with their Advanced plan.

Another site we found was Pingability. The general look and feel of the site is OK, but the service offered was not great. The free plan requires a back-link (which we think is unacceptable for a professional site). At the same time the premium service, for $9.95, only offers one monitor.

Next up for review is Wormly. Priced at $9 per month, their Bronze-plan seems to be a reasonable alternative. The plan includes 5 monitors and they ping your server 5 times every 5 minutes, which is good enough. Unfortunately there’s a big ‘but’ — no SSL monitoring (at least as far as we can tell). That’s a deal-breaker. To Wormly’s defense though, they do offer something that sets them apart from the competition, namely the ‘Server Health Monitor.’ This service is something similar to Cacti (it definitely looks RRDTool-based), that visualizes server-load. However, they will probably have a hard time selling this service to security-concerned organizations, as they require a monitoring-client to be installed on the server (it’s hard to get this data otherwise).

Basicstate is the final service we will cover in this article. A lot can be said about Basicstate’s web design (it’s _really_ bad). However, they do offer a very competitive service. They ping every 15 minutes and allows you monitor as many sites as you want (including SSL). While it might not be a very pleasing site to browse, they do offer sufficient statistics (with graphs) on their site. In addition to that, they also send you daily reports about all your monitored sites (with time data for dns, connect, request, ttfb, ttlb). The only drawback we discovered with Basicstate is that you cannot monitor the same domain-name with SSL and non-SSL (sub-domains is fine though). This may or may not be an issue for you.

The verdict? We settled for Basicstate. Later on, as we grow, we might consider switching to Pingdom. We’re happy with Basicstate for now. Although we did experiencing some false alerts, the guy who runs the site (I assume), Spenser, did a great job on providing an in-depth explanation to the alerts by email. So if you’re on a tight budget, Basicstate is our recommendation. If you have more money to spend, go for Pingdom.

Author: Tags: ,

© 2006-2009 WireLoad, LLC.
Logo photo by William Picard. Theme based on BlueMod © 2005 - 2009 FrederikM.de, based on blueblog_DE by Oliver Wunder.
Sitemap