We’ve recently built a realtime web-app product. It allows teams to communicate in chat areas, write shared notes, upload documents, poll team members, among other things. The engine we chose to run and store all the data was Twilio Chat because it provides an easy way to ensure all connected clients are synchronised and are showing the same data.
We finished the MVP, launched the product and watched it work exactly as we intended.
However, as you can see from the title of this post, we had to migrate away from Twilio Chat.
We intend to sunset the Programmable Chat API on July 25, 2022 to focus on the next generation of chat, the Twilio Conversations API. 1
When we first discovered Twilio was replacing Chat with Conversations we were not so worried as they seemed similar. But, after closer inspection, Conversations has a couple key differences that essentially prevent our use case. Namely, all Conversations are private, in order for a new member to join an existing Conversation they need to be invited by an existing member.
We were faced with a choice: change the logic of the whole app or migrate to another service that supported our use case. Since we were also not very happy with the backup system Twilio offers to download messages and the fact that each channel can only have up to 1000 members, plus, ultimately, we felt disappointed and untrusting of Twilio (who’s to tell they won’t sunset Conversations in a couple of years), we were more inclined to move to another service.
After researching and testing a couple of options, we ended up choosing PubNub. They recently changed their pricing to a Monthly Active User system (similar to Twilio) making the monthly cost similar.
Since we have a working live product we also wanted to make the migration as smooth as possible, ideally making it gradual so we can catch issues before they affect everyone.
Looking at Twilio Chat source code we realised it is a chat-specific sugar syntax on top of Twilio Sync (Twilio’s Real-time state synchronisation API). Also, while PubNub is pushing their “in-app chat” features, they are in fact closer to Twilio Sync than Twilio Chat. So our migration code would have to create the missing layer.
Twilio Chat vs PubNub [internals]
Twilio Chat is based on instances of a Client, Channel, User, Message, Media. Each instance can emit and listen for events and is connected to each other (a Message has a Channel, and a Channel has Messages). That means instances update their internal state automatically and since we are using Vue.js that triggers a change in the component (yes, it feels like magic).
PubNub, on the other hand, has a centralised “PubNub Object” that can subscribe and publish to channels, add event listeners, fetch messages and retrieve user/channel metadata objects.
This means Twilio Chat and PubNub follow fundamentally different paradigms and we would not be able to easily switch from one to the other.
The migration code itself
What our migration layer does is it adds the concept of a Client, Channel, User, Message, Media classes. For example, when the PubNub “message” event is triggered we create a new instance of a Message, store it in the respective Channel instance and emit the “messageAdded” event on the Channel.
The other important aspect of the migration layer is to convert the PubNub naming scheme to the Twilio Chat names: “uuid” becomes “author”, “timetoken” is “dateCreated” and “sid” (PubNub indexes messages by timetoken instead of assigning a unique random id).
By providing a translation layer we can continue to call the same methods from Twilio Chat and then convert them to the PubNub equivalent in our custom migration layer.
An interesting challenge we had was how to perform message updates. With Twilio Chat it is possible to push an update to an existing message and it will be synced across all connected clients.
However, “PubNub stores all messages in a time-series database, it doesn’t expose an API to directly update a message” 2. The way message updates work on PubNub is by sending a message action that works like an attachment to the original message. There is a specific “messageAction” event that triggers when a new message action is added and each connected client can merge the content of the message action into the current state of the message.
The positive outcome of this is that there is a record of all the changes made to the message, and clients can push just the attributes they want to change on the message (instead of the full attribute object as it was required on Twilio Chat).
We are still in the process of testing everything and getting ready to pull the plug on Twilio Chat. Most of the client side code is done and working. By using a migration layer we essentially only need to update the entry-point file where we were initialising Twilio Chat and call our own “PubNub Chat” instead. Since they have the same API, none of the components need to be updated.
We are also looking forward to migrating the transcripts system since PubNub provides an API to fetch messages with much more granular parameters.
Anyone else had to do a similar migration? Let us know how it went on twitter @whitesmithco.