How we do SEO in Meteor
SEO for a framework that denies its existence
If you’ve ever encountered SEO in dynamic Javascript applications, you know that it can be a pain in the ass. Meteor, of course, is no exception. Meteor, in fact, is the one that makes it even worse with its DDP and all sorts of dark magic. It’s way of doing everything automagically makes you think it’ll fix everything, and you can forget about fundamentals, such as SEO. As you probably know, most search engine crawlers don’t speak javascript, so when they try to load your shiny Meteor application, they’ll probably be greeted with something like this:
Yes, that’s a blank screen. No beautiful frontend, no DDP connections to the backend, no metadata. Googlebot is a little bit more advanced, they’ll try to parse your complex application, but they’ll probably hang at the loading screen. At least they do on our site.
As you can see, SEO in Meteor is that pizza you ordered but never came. Meteor Development Group (MDG) offers no apparent solution to this problem, and it seems that they don’t care about it enough. You can get free SEO support through prerender.io using Galaxy, but if you want to host your own Meteor application, you’re left on your own.
The idea
As a developer, you know that organic traffic is rather important for overall growth of the application, so we had to find a way to fix this. Running without SEO is a bad idea. Our use case is quite interesting, since we have lots of Javascript pages with dynamic content, and we have to handle internationalization, as our content is available in multiple languages.
One obvious solution was to intercept all connections to our application, check the user agent, and serve our content accordingly. Namely, regular clients would get an unmodified version of our application, with full blown Meteor code, and search engine crawlers would be served with a special, modified page, that contains all expected data, but without any javascript, only pure HTML pre-rendered on our server. After some consideration, we’ve decided to go with that idea.
The implementation
Server side routing
Detecting user agents in Meteor is a simple task. With help from Picker, a Meteor server side router smart package, we were able to filter all connections and route them according to their user agent.
Here’s a simple filter implementation:
You can modify this and add a lot more search engine crawlers, but this small list is enough to begin with.
This basically intercepts all connections to the Meteor application, and routes only the ones matching the filter lambda function.
Now, you can route bots wherever you want.
Routing with Picker is simple enough:
As you would expect, request and response are regular NodeJS Request and Response objects, respectively.
Server side rendering
Now, when we can route bots to our modified pages, we still have to serve them with something. Otherwise, they would still get a blank page, and that’s what we are trying to avoid.
This is where SSR (server side rendering) in Meteor comes in handy.
Basically, we can define modified templates for search engine crawlers, and render them on our server, just like the browser would do on the client. We can use regular template helpers and even Blaze components.
This sounds great, doesn’t it? Here’s some code to fill up the blanks.
This small example shows how we basically do it.
The problems
Now, the most interesting part, the problems we’ve encountered while working on the solution.
Internationalization
Welcome to internationalization hell, our first, and biggest problem. Since we handle all translations with TAPi18n, which is a purely javascript solution, we had to find a way to let the search engines know that we have multiple versions of our pages, in multiple languages. (currently, only Serbian and English)
Here comes the :lang parameter.
Remember the example above, here it is with internationalization support.
Of course, we’ve made sure that our sitemap.xml also has these routes with :lang parameter, so search engines could find them. For sitemap generation, we are using a sitemap package by gadicohen. It’s easy to use, it has a simple syntax, and it generates valid xml sitemaps.
Here’s a simple example code used to generate sitemaps:
Regular users are still served without the additional :lang parameter in the URL, and if the :lang parameter somehow gets to the client, our client side router, Iron Router, just ignores it.
Markdown support
Markdown support in our pre-rendered templates was also a big concern. Since we couldn’t use the #markdown template helper, we had to register a new one.
Adding this code to the server side router enabled us to use Markdown wherever we wanted. For Markdown parsing, we are using marked package, it does wonders.
Optimizations
Server side rendering is hard on the CPU of the server, and shouldn’t be used extensively. Since multiple search engines make quite a lot of requests, we’ve decided to cache our prerendered templates, and therefore speed up the whole process and reduce stress on the server. For caching purposes, we’ve used Redis, which handles the task perfectly. IORedis package is in charge of communication with the Redis server, and it achieves good performance. Every page is cached separately, and the key is generated according to URL parameters and the language. So, for example, the key for URL /p/:slug/:lang? would be something like p-${params.slug}-${params.lang}.
By default, we set cache refresh period to 7 days, but it can be modified for any given page, so we have absolute control over overall cache freshness. For the eviction method, we are using LFU (Least frequently used), as it fits our use case to keep pages that are crawled more cached. By implementing Redis caching mechanism, our loading times (for search bots) decreased quite a lot, and CPU usage dropped. Keep in mind that without cache, CPU usage and delivery times grow exponentially, while with Redis they stay pretty much the same at scale. The only suggestion we can make is to scale up the memory for Redis in case you cache, like us, thousands of pages of HTML. Here’s how that looks in a totally relevant chart we made in LibreOffice. Totally relevant.
Lastly, here’s some code on how to implement caching. Of course, we’re still using the previous example.
Alternate solutions
Since this is a fairly common issue with dynamic Javascript applications, there are even services that address this issue. One of the easiest solutions, as I’ve mentioned in the beginning, is definitely using prerender.io, a simple service that acts as a middleware between your website and search engine crawlers. It simply prerenders your page on their cloud servers and sends raw html to the crawler, which then parses all relevant data from your page. This solution is simple enough, but it’s not free.
Of course, given that we have a lot of dynamic pages that would all have to be prerendered, and because of internationalization support, prerender.io is simply not good enough for our scenario. In-house solution is also better because we can control what gets served to the search engine bots, so we can further optimize the pages and improve SEO performance.
We’re looking for a match (no pun intended)
SEO in Meteor is hard, but it’s essential to driving organic traffic and getting your content to the people, so make sure you give it some time. In a lot of ways Meteor is the best thing that ever happened to us, regardless of SEO issues.
We’re building a School 2.0 platform to enable everyone on the planet to get free and open crowdsourced technical education. If you’re interested in changing the world with us and developing interesting solutions to fun, unexpected problems, see our open positions.
