How Goldsky Democratizes Streaming Data for Web3 Developers

Goldsky is a real-time data platform for the Web3 ecosystem. Our platform provides blockchain indexing, subgraphs and data-streaming pipelines that help developers build powerful dApps (decentralized applications). We support more than 10 of the most popular blockchains as sources, as well as ad-hoc user-defined data sources, and we’re used by some of the biggest protocols and networks to power analytics and APIs.

To simplify things for those not in the Web3 world: Our customers fundamentally want to get specific data out of different blockchains, which is not easy to do. Typically, you have to go through various remote procedure call (RPC) providers and APIs, which is a hard, painful and slow process, essentially analogous to old-school API crawling. Goldsky ingests data directly from blockchains, with embedded tools our customers use to build real-time data pipelines that power their apps. We push data directly into our customer’s datastores, allowing them to own the data and configure it for their own use case.

Goldsky’s pipeline details interface

As an example, one of our customers is a popular betting website that tracks real-time betting trends. The bets and auctions it tracks are happening on the blockchain layer to ensure transparency and fairness. We ingest the data from the blockchain, decode it to a form that’s useful, transform it in real time and sync it directly to the customer’s database. This is then used for its API layer to power things like betting order books and statuses.

Let’s take a closer look at the streaming data architecture that powers Goldsky’s platform.

Goldsky’s Streaming-First Architecture

There has been a paradigm shift in the industry recently where people are realizing that you can use data platform technology that was previously used by internal analytics teams to power customer-facing features. The sort of data pipelines that previously only supported reporting and dashboards are now supporting web application functionality.

The trouble is, most app developers are not deeply familiar with the underlying technology stacks for data pipelines, which typically include complex technologies such as Apache Kafka and Apache Flink. And this is where platforms like Goldsky come in, to help democratize developer access to streaming data by providing simple API-driven tools that bring real-time data into applications. These data pipelines as a service allow app developers who don’t know the first thing about managing Apache Kafka to have streaming data at their fingertips.

Streaming data is, of course, fundamental to our stack. In fact, we have a “streaming first” architecture powered by Redpanda (a simpler, faster and more durable Kafka-compatible platform) as our primary streaming-data solution, with Apache Flink used for stream processing. We use Redpanda’s managed service, Redpanda Cloud, to ensure minimal overhead.