Solving SEO+Dynamic Content problem the old school way

03 Dec 2015

Problem statement

1. Serving SEO and dynamic data from the same system.
2. Having a microservices based architecture to supply modules/widgets to the main application.
3. Using the latest technology and frameworks to improve user interactions and enable ease of development so that features can be rolled out very quickly.
4. The service based architecture should also be able to serve the crawler and the user coming via SEO traffic should have the same experience in terms of design and features as the user coming via directnav.

Roadblocks:

We are all aware that howsoever fast the technology and frameworks have matured the problem of Google crawler not being able to index dynamic data remains intact till date. What this means is basically having a separate set of design and user flows for the traffic coming via Google and the directnav traffic. This approach undoubtedly has a lot of drawbacks like low key user experience for all the free traffic coming via Google, rich feature set not being able to get a proper audience, most probable two set of teams working on the same page but using different technology stack, etc.

All of the above mentions roadblocks exist because Google is not able to understand/index the dynamically templates via javascript.

Solutions:

1. Do away with the usage of any kind of JS based frameworks and only work with the server side technologies to generate the templates and the required user behaviour. This solution has been in existence in the industry for a long time now but what it does to an application is what we are all aware of, we end up having  a lot of legacy code in our system and also we are not able to utilise fully the the processing power of modern browsers, our server calls become very high and we end up serving a less interactive application to the user.

2. Having an architecture where we serve static content to the crawler with the exact same design and after serving the static content we somehow trigger
the dynamic application which then takes over and embeds user interactive features on the application. Currently this kind of structure is known as an
isomorphic application, that being said “isomorphic application” is just an idea with multiple implementations existing in the market and new frameworks
coming up claiming the support for isomorphism.

image

Goibibo Implementation for serving User Generated content both for SEO and dynamic pages

As part of Goibibo ecosystem we have a team which handles all the User Generated content for Goibibo which includes reviews,images,comments,likes,questions,answers ,etc and this being a new ecosystem is continuously evolving at a very fast pace with features getting rolled out at a very rapid pace. Now we all know that Google just loves user generated content and it is a very genuine source for improving your SEO ranking among many other factors.

The existing implementation for the UGC ecosystem was is on MEAN stack with Angularjs being the Frontend framework for all the templating as well user interactions but Google is currently not equipped to handle indexing of angularjs dynamic templates.

What we did was we created a service which serves a basic template rendered via Jade with the same design and CSS which is served to the crawler and a self executing script at the end which after rendering the static content triggers the angular application which embeds all the user interactive features for
the application.

image

Implementation Details

1. The first and the foremost requirement for the solution #2 to work seamlessly was to give the control to trigger the application to the page rendering the
application, for this we created a static array and push all the controller and directive function to that static array and created an init hook which when called
pops all the controllers and directives and executes them which is basically the triggering of the application.

2. We served a single combined minified file for all our javascript via CDN as part of JSON response and did the same for CSS which got embedded on the page via
the server.

3. we served two set of HTMLs as part of JSON response, one for the crawler and the other for the adynamic application.

Sample response;

http://ugc.goibibo.com/api/HotelReviews/getSeoHtml?vid=2408530086566891690

image
image