A few weeks ago, we spoke at OpenSearchCon India about how we make search fast and reliable at scale at Blinkit. I’m Harshit , from the Search Engineering team at Blinkit, and this year we got to speak at OpenSearchCon India at the Jio World Convention Centre in Mumbai. In this post, I’ll walk you through our talk, the conversations…
#data
55 posts
24 Jul
14 Jul
Expedia Group Technology — Innovation A framework for how we build, deploy, and evolve AI systems for impact and scale Photo by Florian Wehde on Unsplash There’s an important distinction between Artificial Intelligence (AI) that just works today and AI that lasts at scale. Many companies optimize hard for the first one without ever asking whether they’re building the second.…
10 Jun
At Spotify, data problems used to follow a specific pattern. You'd look for the relevant dashboard, there... The post Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant appeared first on Spotify Engineering.
18 May
TL;DR LLM evals, automated judges that assess relevance, coherence, and quality at scale, are a powerful new... The post Better Experiments with LLM Evals — A funnel, not a fork appeared first on Spotify Engineering.
13 May
Moving from an internal tool to a community-driven, production-ready data mesh. By : Ryan Tanner , Raymie Stata , Adam Miskiewicz Introduction We’re excited to announce the 1.0 release of the Viaduct. This release marks a shift from Viaduct being an Airbnb-internal tool that happens to be open source to a true community-driven project with a stable public API. The…
28 Apr
Your LLM issues are really data issues
Stack OverflowRyan welcomes Harsha Chintalapani, co-founder and CTO at Collate and co-creator of Open Metadata, to the show to discuss why AI and LLMs struggle with real-time, structured production data. …
22 Apr
Background Coding Agents: Supercharging Downstream Consumer Dataset Migrations (Honk, Part 4)
SpotifyHow we used Honk, Backstage, and Fleet Management to ease the pain of migrating thousands of datasets. The post Background Coding Agents: Supercharging Downstream Consumer Dataset Migrations (Honk, Part 4) appeared first on Spotify Engineering.
14 Apr
Discover how Airbnb prioritizes user privacy while building a more connected community, empowering guests to engage socially, connect confidently, and maintain control of their personal data. By: Joy Jing ✨ Building a more connected community At Airbnb, our hosts and guests form the heart of our community. As shared by CEO Brian Chesky , we’re evolving into a more social…
5 Mar
Kenza Boulisfane, Software Engineer at Thumbtack, works at the intersection of data, AI, and real business impact. In this Q&A, she shares how she’s building an AI-powered Marketing Analytics Agent designed to make complex marketing data accessible to everyone. She also reflects on team culture, technical challenges, and why diverse perspectives make engineering stronger. What are you currently working on?…
3 Dec 2025
Announcing the preview of open, modular data ingestion building blocks in .NET, empowering developers to build scalable AI pipelines with seamless integration, extensibility, and easy getting started experiences across the .NET ecosystem. The post Introducing Data Ingestion Building Blocks (Preview) appeared first on .NET Blog.
5 Nov 2025
Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration
HerokuModern businesses don’t just run on Salesforce—they run on entire ecosystems of applications. At Heroku, we operate dozens of services alongside our Salesforce instance such as billing systems, user management platforms, analytics engines, and support tools. Traditional approaches to unifying this data create more problems than they solve. In this article, we’ll see how we […] The post Building an…
6 Mar 2025
Discover how Big Data in Healthcare is enhancing patient care, efficiency, and innovation, while securing sensitive data. The post Understanding Big Data in Healthcare appeared first on Erlang Solutions.
22 Aug 2024
Data Residency Compliance Is Possible with the Right Cloud Provider Because today’s companies operate in the cloud, they can reach a global audience with ease. At any given moment, you could have customers from Indiana, Indonesia, and Ireland using your services or purchasing your products. With such a widespread customer base, your business data will […] The post What is…
15 Aug 2024
Modern applications have an unceasing buzz of user activity and data flows. Users send a flurry of one-click reactions to social media posts. Wearable tech and other IoT sensors work nonstop to transmit event data from their environments. Meanwhile, customers on e-commerce sites perform shopping cart actions or product searches which can bring immediate impact […] The post Building an…
7 Aug 2024
At Clever we move a lot of data, both from school districts into Clever and from Clever into all the external places that school districts want that data shared. Broadly, we call this behavior “syncing”, and various sync flavors show up across Clever. Let’s talk about two ways to perform data syncs by using an […] The post Everything but…
7 May 2024
Announcing Data Wrangler: Code-centric viewing and cleaning of tabular data in Visual Studio Code
Microsoft Python EngineeringToday, we are excited to announce the general availability of the Data Wrangler extension for Visual Studio Code! Data Wrangler is a free extension that offers data viewing and cleaning that is directly integrated into VS Code and the Jupyter extension. It provides a rich user interface to view and analyze your data, show insightful […] The post Announcing Data…
28 Mar 2024
Introduction Heroku is excited to introduce nine new dyno types to our fleets and product offerings. In 2014, we introduced Performance-tier dynos, giving our customers fully dedicated resources to run their most compute-intensive workloads. Now in 2024, today’s standards are rapidly increasing as complex applications and growing data volumes consume more memory and carry heavier […] The post Expanded Memory…
31 Aug 2023
In May 2023, we announced our limited release of two new Heroku Private Spaces regions: India (Mumbai) and Canada (Montreal). This month, we’re announcing the full general availability of those two regions, along with new Heroku Private Spaces regions for the United Kingdom (London) and Singapore. This expansion enables customers to maintain greater control over […] The post Heroku Private…
7 Mar 2023
Make your reports faster: A beginner’s guide to Tableau Optimisation In today’s world, given the pace at which data operates, we need a tool that can help us to generate reports faster and bring out insights within milliseconds. In order to solve this challenge, several companies have started utilising a few Business Intelligence (BI) tools such as Tableau/Power BI/Superset/Looker/Qlikview, etc.…
17 Feb 2023
2022 was a transformational year for Heroku. In this post, we share how we’ve been enriching the Heroku developer experience in 2022, especially since committing to Heroku’s Next Chapter. We are dedicated to supporting our customers of all sizes who continue to invest and build their projects, careers, and businesses on Heroku. Public Roadmap As […] The post Heroku 2022…
23 Mar 2022
We launched Salesforce Functions last fall and the response so far has been terrific. While the most obvious use cases for functions are stateless processing of data, there are many examples of business processes that can take advantage of the simplified operating model of functions, but require some persistent state to span function invocations. Today, […] The post Heroku Data…
9 Nov 2021
6 Oct 2021
In a world where data and AI are reshaping society, people currently have no tangible way to put their data to work for the causes they believe in. To address this, we built the Rally platform, a first-of-its-kind tool that enables you to contribute your data to specific studies and exercise consent at a granular level. Mozilla Rally puts you…
20 Jul 2021
As we’re all aware by now, we made some big platform changes at the end of 2020. Whilst the big move has happened, it’s given us a great opportunity to clear out the cupboards and closets. The post Spring Cleaning MDN: Part 1 appeared first on Mozilla Hacks - the Web developer blog.
29 Apr 2021
It seems like a simple enough concept: You take data from how your users interact with your product, and you use it to make business and…
16 Oct 2020
“To me, legacy code is simply code without tests.” — Michael Feathers If untested code is legacy code, why aren’t we testing data pipelines or ETLs (extract, transform, load)? In particular, data pipelines built in SQL are rarely tested. However, as software engineers, we know all our code should be tested. So in this post, I’ll describe how we started…
8 Oct 2020
This summer, we announced the beta release of our new streaming data connectors between Heroku Postgres and Apache Kafka on Heroku. These connectors make Change Data Capture (CDC) possible on Heroku with minimal effort. Anyone with a Private or Shield Space, as well as a Postgres and an Apache Kafka add-on in that space, can […] The post Heroku Streaming…
27 Jul 2020
10 Jul 2020
Today we are announcing a beta release of our new streaming data connector between Heroku Postgres and Apache Kafka on Heroku. Heroku runs millions of Postgres services and tens of thousands of Apache Kafka services, and we increasingly see developers choosing to start with Apache Kafka as the foundation of their data architecture. But for […] The post Streaming Data…
11 Jun 2020
We are thrilled to announce that Heroku Shield for Redis is now generally available and certified for handling PHI, PII, and HIPAA-compliant data. Heroku Shield for Redis is the final missing data service for Heroku Shield, which is an integrated set of Heroku services with additional security features needed for building high compliance applications. All […] The post Heroku Shield…
6 May 2020
Security is always top of mind for Heroku customers; COVID-19 has further increased the urgency for enterprises and developers to deliver more mission-critical applications with sensitive and regulated data. Given the needs of our customers, including those in regulated industries like Health & Life Sciences and Financial Services, we are thrilled to announce that Heroku […] The post Bring Your…
3 Mar 2020
The topic of datasets is not new, and they have been successfully used at SoundCloud for analytics purposes for a long time. In this blog…
4 Feb 2020
After a successful evaluation period, PostgreSQL 12 is now the default version for new Heroku Postgres databases and an available upgrade for existing databases. I want to emphasize a few key changes and improvements in Postgres 12: Native Table Partitioning Concurrent Operations Native Partitioning was introduced in PostgreSQL 10 and performance improvements for improved parallel […] The post PostgreSQL 12…
1 Oct 2019
We are thrilled to announce that Apache Kafka on Heroku Shield is now generally available and certified for handling PHI, PII, and HIPAA-compliant data. Our newest managed data service unifies Heroku Shield, a set of Heroku platform services that offer additional security features needed for building high compliance applications, with Apache Kafka on Heroku, our […] The post Apache Kafka…
23 Jul 2019
There are many reasons to choose Heroku Data services, but keeping the services you use secure and up-to-date rank near the top. This foundation of trust is the most important commitment we make to our customers, and frequent and timely maintenances are one way we deliver on this promise. We do everything we can to […] The post Why Frequent…
16 Apr 2019
The data we store holds value, but refining data into meaning remains a difficult task. Over the last few months, we’ve taken a step back to figure out what we can do to help our users cross that divide, and rebuilt Heroku Dataclips from scratch with that goal in mind. The result is an experience […] The post A Dialog…
6 Dec 2018
Building a SaaS product, a system to handle sensor data from an internet-connected thermostat or car, or an e-commerce store often requires handling a large stream of product usage data, or events. Managing event streams lets you view, in near real-time, how users are interacting with your SaaS app or the products on your e-commerce […] The post Manage Real-time…
11 May 2018
Track play counts are essential for providing a good creator experience on the SoundCloud platform. They not only help creators keep track of their most popular songs, but they also give creators a better understanding of their fanbase and global impact. This post is a continuation of an earlier post that discussed what we do at SoundCloud to ensure creators…
19 Apr 2018
For data engineers and analysts, it’s pretty common to get questions about missing or incorrect data. “Hey Data Engineer, there’s an issue with the data – I expect numbers at least 20% higher than what our reporting tools show. Can you take a look?” If you’ve ever been responsible for a Business Intelligence pipeline, you’ve […] The post Save sanity…
24 Jan 2018
SoundCloud consists of hundreds of millions of tracks, people, albums, and playlists, and navigating this vast collection of music and personalities poses a large challenge, particularly with so many covers, remixes, and original works all in one place.
19 Dec 2017
Designing scalable, fault tolerant, and maintainable stream processing systems is not trivial. The Kafka Streams Java library paired with an Apache Kafka cluster simplifies the amount and complexity of the code you have to write for your stream processing system. Unlike other stream processing systems, Kafka Streams frees you from having to worry about building […] The post Kafka Streams…
26 Oct 2017
Today we are pleased to announce a significant update to Heroku Connect, one that is a culmination of two years of work to improve every aspect of the service. We’ve focused on three primary areas: improving write speed, geographically expanding the service, and intelligently guiding design and troubleshooting workflows. To that end, we’ve enabled bulk […] The post Heroku Connect…
4 Oct 2017
Here at SoundCloud, we’ve been working on helping our Data Scientists be more effective, happy, and productive. We revamped our organizational structure, clearly defined the role of a Data Scientist and a Data Engineer, introduced working groups to solve common problems (like this), and positioned ourselves to do incredible work! Most recently, we started thinking about the work that a…
14 Sept 2017
Event-driven architectures are on the rise, in response to fast-moving data and constellations of inter-connected systems. In order to support this trend, last year we released Apache Kafka on Heroku – a gracefully integrated, fully managed, and carefully optimized element of Heroku’s platform that is the culmination of years of experience of running many hundreds […] The post Kafka Everywhere:…
20 Jun 2017
Once upon a time, we had a single monolith of software, one mothership running everything. At SoundCloud, the proliferation of microservices came from moving functionality out of the mothership. There are plenty of benefits to splitting up features in this way. We want the same benefits for our data as well, by defining ownership of datasets and ensuring that the…
11 Jan 2017
At Heroku, we’re always working towards improving operational stability with the services we offer. As we recently launched Apache Kafka on Heroku, we’ve been increasingly focused on hardening Apache Kafka, as well as our automation around it. This particular improvement in stability concerns Kafka’s compacted topics, which we haven’t talked about before. Compacted topics are […] The post Pulling the…
29 Nov 2016
Heroku recently released [a managed Apache Kafka][1] offering. As a Node.js developer, I wanted to demystify Kafka by sharing a simple yet practical use case with the many Node.js developers who are curious how this technology might be useful. At Heroku we use Kafka internally for a number of uses including data pipelines. I thought […] The post Data Pipelines…
28 Sept 2016
Many of the compelling and engaging application experiences we enjoy every day are powered by event-based systems; requesting a ride and watching its progress, communicating with a friend or large group in real time, or connecting our increasingly intelligent devices to our phones and each other. Behind the scenes, similar architectures let developers connect separate […] The post Apache Kafka…
10 May 2016
Today we’re announcing that the APIs for the Heroku Connect data synchronization service are now GA. These fully supported endpoints will help our users with the tasks they most need repeatable automation for: creating consistent configuration across development, staging, and production environments; managing connections across multiple Salesforce deployments; and integrating Heroku Connect status with their […] The post Heroku Connect…
26 Apr 2016
Today we are happy to announce early access to Heroku Kafka. We think Kafka is interesting and exciting because it provides a powerful and scalable set of primitives for reasoning about, building, and scaling systems that can handle high volumes and velocities of data. Heroku Kafka makes Kafka more accessible, reliable, and easy to integrate […] The post Announcing Heroku…
25 Jun 2015
Today we’re pleased to announce general availability of Heroku Redis with a number of new features and a more robust developer experience. By giving developers a different data management primitive, we’re helping them meet the needs of building modern, scalable applications. The classic example of using multiple data stores in an application is the e-commerce […] The post Heroku Redis…
12 May 2015
Developers increasingly need a variety of datastores for their projects — no one database can serve all the needs of a modern, scalable application. For example, an e-commerce app might store its valuable transaction data in a relational database while user session information is stored in a key-value store because it changes often and needs […] The post Heroku Redis…
30 Jul 2014
In May we released the first version of Heroku Connect, a service that makes it easy to build Heroku apps that share data with your Salesforce deployment. Today we released our first major update to the service, bringing new speed and scale enhancements to all Heroku Connect users. Together, these enhancements lower latency on Heroku […] The post Heroku Connect:…
3 Jul 2014
Here at SoundCloud, in order to provide counts and a time series of counts in real time, we created something called Stitch. Stitch was…
5 Jul 2011
MySQL turns out to be a good Swiss Army Knife for persistence, if used wisely. Understanding disk access patterns driven by your storage…