Accept Incoming Emails into a Heroku App Using SendGrid

I love me some Heroku.  If you’ve never heard of Heroku, it’s an awesome Ruby on Rails hosting company.  It makes deploying and hosting rails apps an absolute breeze.

Anyway, I wanted my newest creation, www.SaleTrap.com, to read incoming emails and respond to them.   After poking around the Heroku mailing list, and reading the documentation, I ran across this – http://wiki.sendgrid.com/doku.php?id=parse_api, but no step by step instructions on how to get it working.   So without further ado, here’s a step-by-step guide on how to get your Heroku rails app to read incoming email.

  • Create your rails app on Heroku.  Duh.
  • Add the sendgrid add on to your app.
  • Create a controller to read the incoming email post.   For example:

  • Now starts the fun part.  Fire up your command line, and go to the /app directory of your app.  type in “heroku config –long”.  It will return your sendgrid username and password
  • Awesome, now venture over to www.SendGrid.com and login with those credentials.  Go to the “develoeprs” tab, then the “parsing incoming” sub tab  (url will be http://sendgrid.com/developer/reply)

  • Alright.  Time to get tricky.    The first part is easy – set the url to your controller action above.
  • The second part is a bit more complex – you can’t specify a single email address in sendgrid, so you have to have all emails for your domain, or a chosen subdomain hit the parse api.   In this example, I’ve chosen “watch.saletrap.com”.
  • Once you have your subdomain, enter it into sendgrid.  Phew, your work is done here.
  • Now, on to your DNS provider.   I use GoDaddy, but this should work with whatever.   Go to your DNS control, and enter a MX record, using your subdomain (i.e. “watch” in this case), as the host, and “mx.sendgrid.net” as the destination.

  • Still with me?  Good.  Now on to the email address.  I set up an email forwarding account for SaleTrap.com with GoDaddy, with the “watch@saletrap.com” address, that points to watch@watch.saletrap.com.   Get the key point there?  The email points to the watch.saletrap.com, and we have an mx record for “watch” that points at sendgrid, so hence, our emails magically get sent to sendgrid, which posts pack to the url in our rails app that we specified.

  • And there you have it, you can now accept incoming emails into your rails app using heroku and sendgrid.

Want to see it in action?   Do the following:

  1. Go to www.JCrew.com, and choose an item you like.
  2. Copy the url of that item (e.g.http://www.jcrew.com/AST/Browse/MensBrowse/Men_Shop_By_Category/shoes/casualbootschukkas/PRDOVR~20525/20525.jsp), and email it to watch@saletrap.com
  3. Wait 30 seconds, and refresh your inbox.   Enjoy

DISCLAIMER:  www.SaleTrap.com is far from complete yet, so don’t expect too much at this point.  It’s still under active development, so will probably be buggy.

I want to send a HUGE thank you to Isaac at SendGrid for helping me out.  He, and SendGrid, rock.

About these ads
This entry was posted in Uncategorized. Bookmark the permalink.

49 Responses to Accept Incoming Emails into a Heroku App Using SendGrid

  1. Awesome use of both Heroku and SendGrid. The forward from watch@saletrap.com to watch@watch.saletrap.com is a great idea. Please keep us updated as you add more stores, this is a really cool idea!

  2. Ray says:

    Have you tried with parsing the attachments?
    I’ve been going in circles with that.

    • Doug says:

      Actually, I have, for another project. In this example, I am receiving an email with a text file attachment that I need to read some values out of.


      def receive_email
      @params = params
      #make sure request is a post
      text = params["text"]
      html = params["html"]
      to = params["to"]
      from = params["from"]
      subject = params["subject"]
      attachment1 = params["attachment1"]
      lead_string = attachment1.read
      email_leads = lead_string.scan(/\d.*/)[0].to_i
      phone_leads = lead_string.scan(/\d.*/)[1].to_i

      • carl says:

        does the above go into a controller or the model file?

        in the example your processing info is in the controller… what all is in the model, as well as the html file that passes the received email to the controller

      • Doug says:

        The code demonstrates accepting the email with controller code, and saving it to a model (called IncomingEmail). The model has fields to match the incoming email – sender, text, from, etc. I save the html to that model, and then do some processing to figure out what’s in the html itself. You don’t necessarily have to use a model, I do in my application because i want a history of the incoming email.

  3. Pierre says:

    Very nice ! Thanks for this great tutorial.

    How do you deal with email address that look like :
    Pierre Valade

    Do you some kind of regexp ?

    • Doug says:

      All of the emails I have processed so far have arrived with a from value that includes a string name or names, and an email address in the same field, where the name can be a single name (e.g. Fred), multiple names (e.g. Fred Wilson), or something similar to that. I strip out the email by doing some simple ruby string functions:

      email =from.split('')[0].downcase.gsub(' ', '')

      I am sure there are better ways to do it, but it works for now.

      • mfoley23 says:

        I tried using your code above to strip out the email only from the “from” field and wasn’t able to get it to work… I’m trying to grab the email address of the sender as it’s coming in and check against my user database to make sure they are a user, so the email address has to be exactly right for the trick to work. Any thoughts?

  4. Phil says:

    Doug;

    Thanks for this post, it (kind of) helped us.

    Our app, instead of having a single incoming address like yours seems to, gives each user their own email address. They forward certain emails to their address, and they are parsed. Heroku hosted. I love me some heroku, too. :)

    Oddly, sendgrid seems to ignore any email that is forwarded. Only ones composed from scratch seem to get parsed. Did you experience anything like this? And are you still using sendgrid to parse your emails? We really need it to work. :)

    • Doug says:

      Hmm . . . forwarding works for me. A couple of thoughts:

      • Is your app looking at the from email address and ignoring emails that don’t match an email stored in your system? A forwarded email comes through with the from address sent from the last sender, not the original
      • Is spam monitoring turned on in sendgrid (it’s on the same setup screen)? That could be messing with things.
      • Phil says:

        Hi Doug. I had a lengthy back and forth (friendly) with support at sendgrid. What they told me is that the “To” param in the post from sendgrid when they receive an email does not contain the email address that actually sent the email to them. For example if I set up an address “woohoo@saletrap.com” that forwarded eventually to mx.sendgrid.net, there is no way to reliably get that email address “woohoo@saletrap.com” from the params of the post.

        That is frustrating to us because we went to use that very item (the email address) as they key to determine what happens with the email.

        They said that they intend, in the near future, to fix this, and I am pretty much on hold with my app until they do. thank you for your informative post on this subject.

        Phil

      • Doug says:

        Bummer, it’s frustrating to be on hold. They’ve been quite responsive to my requests, so hopefully they can change this shortly for you.

  5. Steve Smith says:

    Hey I’d love for you to give http://cloudmailin.com a try. We are aiming to make a really simple yet powerful system for allowing incoming email. At the moment we’re in beta and it would be awesome to get some feedback!

    • Phil says:

      Steve I definitely will check our your service and much appreciate the link! I suppose I’ll post the results here unless Doug objects.

    • Phil says:

      Steve;

      Cloudmailin got added to Heroku as an addon! Our app is hosted there.

      I went to add it on but it says it is only available to private beta users.

      How can I use the addon via heroku? We REALLY need this; our app is basically ready to go except for this piece. I’d much rather use cloudmailin than hack together some solution using tools not really intended for the job.

      • Steve Smith says:

        @Phil, we have been working pretty hard on it and we’re really happy to have been accepted on the Heroku addons program. Unfortunately we can’t add people as far as I know as it’s the internal Heroku beta list. However, give me a shout steve@[you know what] and I’ll see what we can do.

  6. Pingback: Accepting emails into a Heroku hosted application | DeveloperQuestion.com

  7. Sen says:

    Doug,

    Isn’t putting ‘skip_before_filter :verify_authenticity_token’ in controller vulnerable to spam?

    Any ideas on how to fix this?

    -Shan

    • Doug says:

      Yes, a better way to do it is on an action by action basis, like:

      protect_from_forgery :except => :index

      That still leaves some vulnerability of someone posting to the form. You could add checks to see what the requesting domain was to make sure it’s coming from sendgrid. I’d love to hear other approaches as well.

    • Steve Smith says:

      Yes it certainly can be. The only way we could think of to fix this was to create a secret key that cloudmailin uses to sign any message it sends users can then choose to either ignore it or validate it. No approach can really prevent a user from forging an email entirely but this at least catches casual spam bots and ensures the email is always coming from the right place.

  8. Pingback: Quora

    • unthinkingly says:

      Quora, That’s exactly what I want to do, send photos via email from my phone. I actually will probably use a 3rd party app, but I’m learning ruby so I’d like to learn — did you write your own, and if so what was your experience?

      I’d appreciate any input on building a smart email image processing app on heroku.

  9. Sid says:

    Was wondering how this process works on FREE Heroku about i.e. our-project.heroku.com doamin. Do we have access to DNS records, i’m guessing we have to have a paid Heroku account with our own URL for this method to work, is that right? We are just trying to get something setup with Heroku and Rails to receive emails, so let me know if there’s a way to test this out easily?

    • Doug says:

      Great question. I’ve always just registered a domain and pointed it to my free heroku app to get it to work. .info domains are only $1.99 per year on GoDaddy. Once you register your domain with GoDaddy (or whatever registrar you want), they will give you access to the domain records.

    • Steve Smith says:

      I’m not sure if this helps but on CloudMailin we faced this problem and decided to issue a unique email address to each user that can be used for testing. Once you’re ready to use it in production you can then add the custom domains addon to use your own domain. Of course I completly agree with Doug that you can’t go far wrong with a 1.99 domain.

  10. Doug says:

    CloudMailin is also a great solution to this problem, I’ve tried it out and it works very well. I should have mentioned that in my response, thanks Steve.

  11. Hey Doug, Thank you for explaining how you configured the godaddy mx record info. This was a bit tricky to understand at first.

  12. Pingback: links for 2011-09-11 « Bloggitation

  13. emailnewbie says:

    This is super helpful. Does anyone know of any ways to test this in development? The only way I’m able to test this is to deploy on heroku and then send a mail to the address that receives, it works for now, but it’s not sustainable…

  14. Doug says:

    Thanks Steve, appreciate the response.

  15. rubynewbie says:

    What’s the purpose of clean_field? I started getting errors from it so I took it out. I see that it’s replacing a new line with a space, do databases have a hard time storing new lines?

  16. here is the send grid documentation to parse incoming email: http://docs.sendgrid.com/documentation/api/parse-api-2/

  17. Sai Vardhan says:

    Hi.,

    Nice work Man. Can You please help in this case. I too have a same requirement regarding this Email. My application is built in rails 2.3.11 and present I am working on the Creating a controller for receiving the parse messages.

    How can I build the model regarding this controller

    Thx.
    Sai

    • Doug says:

      You bet, here’s the model I am using, this should get you started:


      create_table "inbound_emails", :force => true do |t|
      t.text "text"
      t.text "html"
      t.string "to"
      t.string "from"
      t.string "subject"
      t.integer "attachments"
      t.datetime "created_at"
      t.datetime "updated_at"
      t.integer "user_id"
      end

      • Sai says:

        Hi Doug.,
        Thanks for the reply.How did you manage the multiple attachments here. Are you storing the attachments or simply storing the no.of attachments
        here is my controller code

        class ListenerController
        def new
        @test =Test.new(params[:test])

        end

        def create
        @test=Test.new(params[:test])
        if request.post?
        if @test.save!
        redirect_to :action => “new” and return

        end
        end
        end
        and in my model I have placed the fields to,from,body,subject and a attachment field to save attachments. but now I am trying to add the association for attachments is that necessary
        Can I have your maild ID plz

        Thx,
        Sai

      • Doug says:

        Take a look at sendgrid’s parse api for how the attachments will come through . You’ll get the number of attachments, and then each will come through with the number appended to it. See code below for how to read the first attachment, in this case I am reading a text file into a string.


        def receive_email
        @params = params
        #make sure request is a post
        text = params["text"]
        html = params["html"]
        to = params["to"]
        from = params["from"]
        subject = params["subject"]
        attachment1 = params["attachment1"]
        lead_string = attachment1.read
        #do something with these values now
        end

      • Sai says:

        Hi Doug,

        Thanks for the reply. Here is how I managed my controller code
        ######################################################
        class ListenerController “new” and return
        end
        end
        end
        end
        ######################################################
        and test is my model which has to,from,subject,body as attributes and test_file maintains the attachments
        Can this code be acceptable?

        Regards
        Sai

  18. Sai says:

    Hi Doug.,

    I have gone through your controller code. After saving the inbound email to the database, what the method “process _incoming_email ” will do

    I have stucked up at the sendgrid URL , I am testing the code with the help of GUI but when I remove the GUI and save it as URL in the developers of the Sendgrid it is getting a 4XX status error fro the Sendgrid

    I need to receive the data and to store it in the database eventually. please help me in this case.

    Thanks In Advance

    Regards
    Sai

  19. Sai says:

    Hi Doug,

    I got the working functionality regarding the Sendgrid. Can you please suggest me that how you are handling the “attachments” ie., how you are saving them when a message has more than one attachment

    Regards
    Sai

  20. Pingback: Accept Incoming Emails into a Heroku App Using SendGrid | Nances Kitchen | jon's blog.

  21. Nishutosh Sharma says:

    Hi Doug, and all.
    I have read all the possible documentations on Parse API, I wrote code in my controller as you mentioned in the controller example(image). I got some errors. But first of all I wanna ask you, do I have to install Heroku addon of sendgrid on my app to make it receive emails as I already have a seperate sendgrid account which I have configured in my rails app to send emails through sendgrid…

  22. Pingback: PHP Library for Sendgrid Parse Api | Oliver Friedmann

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s