Connexion
How Malachyte solves retail’s cold-start problem with managed real-time AI
What’s the best way to recommend products to little-known users?
We’ve spent our careers trying to solve this problem for major companies like Spotify and Priceline, and it’s why Sidd founded Malachyte, an AI-powered ecommerce recommendation platform. These days, consumers have come to expect content that feels personalized and relevant, and online services competing for their attention have no choice but to do this exceptionally well.
Malachyte was inspired by some unique insights into how advanced AI models, and large language models in particular, could be applied in new ways to old challenges like personalization and recommendations.
As Malachyte set out to win potential customers’ business, we needed secure, scalable, reliable and, above all, leading-edge AI infrastructure to continue building the personalization algorithm we had always envisioned. By utilizing Google Cloud tools like Bigtable and Managed Service for Apache Kafka, Malachyte has been able to help some of its retailers double and sometimes even triple their sales.
This is the story of how we built it, and the ways any founder can use services like these to start deploying AI foundation models in new ways.
How Malachyte lifted sales for their users
For Malachyte, the aha moment was discovering that it could use neural networks with attention mechanisms — the same concept powering large language models — to personalize retail search and product pages. This approach is what enables LLMs to derive meaning from the relative order of items in a sequence, in their case the order of words and syllables in a sentence. When it comes to a retail website or app, what Malachyte wanted to capture was the sequence of customer interactions with the site.
What if we predicted the next thing a user wants on an ecommerce website just like LLMs predict the next word in a sentence?
A pre-GPT language model might have tried to look at a specific sequence of words or even fragments of words (what we now know of as tokens), but those earlier models wouldn’t examine what happens if the words were in the comparable order but weren’t contiguous or were re-arranged. The breakthrough came — in part through Google’s work on transformers — when LLMs gained the ability to understand complex and long-range dependencies within a sequence of items.
This more sophisticated method has delivered dramatic results — both for the proliferation of gen AI in general, and for Malachyte’s application of the technology.
To make this work in practice, Malachyte creates a vector of everything known about a visitor when they arrive on a site. Most users are visiting for the first time, so little is known about them. This is what’s known as the “cold start” problem. The trick is to use every interaction with a user to refine this vector. Each new addition to the vector, like a click or a query, does two things: it drives a prediction about the next thing the user wants, and it provides more information about the user.
Malachyte’s platform then updates the user vector and the prediction at the same time. This not only enhances the understanding of the individual user and their preferences, it also improves the overall model with the anonymized user data. With every inference, the context of both the average and the specific shopper grows.
The company further innovates by not just using attention-based neural networks but combining that with updating user profiles 100 milliseconds at time.
Malachyte’s recommendation and search agents populate the next page’s search results or recommendation carousels based on what users clicked on previous pages.
To be sure, this idea isn’t in itself new. Retailers have long used collaborative filtering recommendations systems to identify similar users and items that required massive sets of interaction history. These models typically required a lot of data, including third-party cookie-based profiles and demographics.
By focusing on the sequence of interactions in a session, retailers can achieve far more personalization — with less required data or spend — than by focusing only on a user’s profile. As a bonus, retailers can now offer their users more privacy by not relying on long-term cookie data.
This works because of the model structure and multimodal vectors that encode everything they know about a user, including browser data, click history and searches. The output, too, is multimodal: The same model can be applied to on-site search product pages, category pages, and add-to-cart carousels.
To make this work, each product in the catalog is embedded into the same space as the user vector, which gets updated and subsequently moves the vector closer to relevant products and further from those that aren’t. The neural network computing the embedding is being continuously trained across retailers who work with Malachyte, improving the quality for everyone. The system effectively becomes a data cooperative with each retailer's user helping make the model smarter for everyone.
A user session represented as a vector in a space of products.
To make this delivery for every user at every inference in 100 milliseconds, Malachite found real benefits in building onGoogle Cloud’s real-time AI stack.
With this system, every behavioral event streams into a Managed Service for Apache Kafka cluster. Rather than queuing for a future training job, each event immediately becomes an update to the user’s profile in Bigtable. The Kafka cluster allows the customer’s front-end to persist, so the user session signals quickly with little worry about how they fit into the user vector.
Bigtable allows Malachyte’s services to look up and update the right user vectors, and it and Kafka operate at the order of 10 milliseconds per step, which allows the entire recommendation loop to complete with no disruption to the user experience.
The three layer real-time AI architecture: a retailer’s website, Malachyte’s AI models and serving front ends, and context management infrastructure.
In addition to a fast core, a second layer of product catalog updates, inventory signals, and retailer-specific dimensional data keeps product data up to date. This flows through Cloud Pub/Sub, which offers globally accessible REST APIs that enable connections retailers can use without deep integration work. Malachyte agents run on Google Kubernetes Engine (GKE), with model inference on Google Compute Engine (GCE).
Continuous ingestion of external data, such as product catalog updates, operates through Pub/Sub’s global messaging system.
With its migration to Google Cloud’s AI architecture, Malachyte demonstrated that production AI inference and training are about more than GPUs and storage. They require real-time continuous learning infrastructure that includes a fast key-value store, a streaming layer, and a managed messaging system, all integrated with the foundation model architecture.
This approach also shows that even a small team like Malachyte’s can have a big impact in an industry. It just needs access to powerful infrastructure and core AI managed services.
Try it for yourself
Looking to shake up your industry or stay ahead of the competition like Malachyte? Try Managed Service for Apache Kafka, Cloud Pub/Sub, and Bigtable. New customers can receive $300 in Google Cloud credits.
