Syatt's Serverless Journey

Moving data between systems is a crucial task in any ecommerce project.  As a full service ecommerce agency one of Syatt’s favourite responses to integration questions in the discovery phase is, “Yes, we can do that.” Offering services to meet the front-end and backend needs of our clients is what makes Syatt a unique agency that doesn’t shy away from custom integrations. In fact, we pride ourselves on it. 

After many builds that did not have pre-built integrations, Syatt realized the need for a data platform that could handle custom integrations with high scalability. In order to achieve this we took a look at what worked with our existing integration approach, what didn’t work, and what we could do with our team of sophisticated backend developers. After a few months our own framework was born and would later be lovingly known as Ethel.

How could we do this better?

Let's take a common use case in the ecommerce world: Syncing large amounts of catalog data. Suppose there are tens of thousands of products that need to be synced to the ecommerce platform. You could process each product in a long running job, but that will take a very long time and is error prone. The other option is to put each job into a queue and process them asynchronously. Better - but now you have introduced complexity into your hosting. Next thing you know, you are adding servers in the middle of the night and commenting on message boards that Rust doesn't belong in the linux kernel. 

How we used to do it

Syatt’s first complex integrations were built using a traditional server architecture using the following stack:

• • Ruby (with Sidekiq)
  • Postgres
  • Redis

Scheduled jobs that would pull data from one source, transform it and send it along to its final destination.  This approach had several drawbacks for the Syatt team:

All custom all the time

Because each project had its own architecture and purpose each system was very customized to the client itself. Code was not as reusable as it could have been. Changes would need to be manually added to each client, duplicating work and slowly building up dreaded tech debt. 

Too many specialized skills

Specialized knowledge is required to host these sorts of systems. We needed to free up time for our developers to work on problems, not solve outages.

Time to pivot

After learning what was not working for us we decided to try a new approach by migrating to a serverless system.

Serverless computing makes sense under the following circumstances:

• Simple applications that can run in a short amount of time. 
• Unpredictable levels of traffic. 
• Jobs that can run in the background.

Almost all of our needs fit nicely into these three points. We can scale it up to any demand if a client has a huge spike in traffic, and just as importantly, we can scale it back down to zero when no resources are needed (like in the middle of the night). 

Takeaways

A huge success. Syatt can build more reliable and customizable integrations faster than ever before. Scalability is baked into the platform, there is no worry that the integration server will crash in the middle of the night and require troubleshooting from an on-call person. 

Here are some of the many ERP integrations we have built for our clients:

• BlueLink
• Visual
• Epicor Eagle
• Mach
• MS Dynamics
• Fox Pro
• Kerridge (K8)
• Pronto
• Erply