FHIR Proof: Call to Action for FHIR Solutions to Prove their Usability

Here is the ammo to ask the important questions regarding your FHIR solution.

An in depth analysis of common FHIR offerings and their performance against real-world scenarios — along with a Call to Action for better product transparency

Disclaimer: I am employed at Google Cloud as a Solution Architect.

**This story has been edited as of January 29th, 2022 to include SMILE CDR’s Benchmarking the organization published after my call to action. And while I disagree about their “numbers often don’t even matter” bit, since we are talking about patients here — not cars and phones — I am still delighted they published despite the hesitations. I wrote this article as a way to make it clear; we cannot operate on just reputations, as some of the largest, most trusted organizations are failing to meet expectations. So in the name of transparency, I want to thank the entire SMILE team and what an incredible reminder we can push ourselves forward as an industry, together. Next up, crucible battles?

After the recent publication of Google Cloud’s Healthcare API Benchmarking White Paper — I did some digging into what other benchmarking exists for FHIR servers. Turns out, this was a harder task than expected. My findings uncovered dismal scale and performance numbers, varying widely across vendors. I also found an overall lack of transparency and documentation for products making me wonder… Enterprises make decisions based on scale, performance, and cost — so where are those numbers for FHIR offerings? How do decision makers know what questions to ask or what expectations to set for their solutions, if there is no documentation to guide them?

So here is my attempt to give enterprises a guide to what questions to ask, what expectations to set, and what your organization may need for FHIR solutions.

My Research and Call to Action: This blog is using public information — most of it being from product documentation or anecdotal, either from users or threads for various products. This means, this is in no way a complete analysis — in fact, I request anyone who can provide public documentation that materially contributes to this analysis to please email it to girlonfhir AT google.com and I will happily update this blog.

FHIR will only be the interoperability standard if it can be used in real-world use cases. Every database row is a life, every extra second could make a difference for someone. At the end of the day… this is why you should care about scale and performance. This isn’t just an industry buzzword, these are real lives and patient outcomes.

So here are some real world scenarios for perspective…

Why Care about Scale?

In a previous blog I discussed the issues with having multiple FHIR servers.

There are multiple complications with trying to use multiple databases, FHIR servers, and/or federating the servers. The heart of the issue is with referential integrity, synchronization/consistency, and most importantly — authentication, user access, and security. Thus, an organization can only (reasonably) use a single FHIR offering across their population.

If an organization sees ten million patients a year, they would create around five billion FHIR resources a year (1 patient = ~500 FHIR resources). With data retention around three years, the organization will need storage that can support fifteen billion resources, or ~30 TB of data.

A FHIR server should be able to scale to terabytes of data and store billions of resources.

Why Care about Performance?

In the same blog I set up a real-life scenario of streaming healthcare data throughput from electronic health records (EHR) (HL7v2 feeds).

If a large system is sending around 8 million HL7v2 messages a day, there would be ~500,000 messages during a peak hour. There are around 15 FHIR resources created per message, this means the organization is ingesting ~2,000 FHIR resources a second streaming in from their EHR.*

*This is the architecture Google’s Healthcare Data Engine is solving for

A FHIR server should be able to ingest/create thousands of resources per second.

The Options

Scale — storage size supported by the service

For scale — I used what the largest implementation of the product had been documented as. If that information was unavailable, I used limits stated on product pages.

Performance — speed of the service

For performance — there are many many API operations that could be tested for various performance needs. I focused on the creation or ingestion of FHIR data — by resources, FHIR bundles, or in bulk — and the throughput of those options.

Note: I believe search may be the most indicative measure of performance at scale, however, due to the unavailability of documentation from other vendors on this benchmarking, I was unable to provide comparisons at this time.

Type of Server

There are two very different classes of products — unmanaged versus managed services. You can do some googling on the difference between the options, however, from a scale and performance point of view — unmanaged is directly related to the amount of infrastructure provisioned, versus managed is heavily reliant on the service provider.

Managed Services

Google Cloud

Scale

As stated in the white paper — Google was tested using a single FHIR datastore with 50 million patient records (26 billion resources) which equals about 60 TB of data.

Performance

Bulk Import — Ingested 10,038 resources a second (95 percentile)

Execute Bundle — Response time at 18 milliseconds per resource, or ~55 resources/second* (95%) from a single client thread

*Can be parallelized to achieve increased resource per second execution

Overall

Meets enterprise requirements for size and throughput.

Azure

Scale

The documentation states the maximum available RUs for their FHIR API is 1 million, allowing for about 25 TB of data assuming 40 RUs per GB.

Performance

Bulk Import — not supported (user has to use the FHIR loader)

*FHIR Loader — 500 resources a second at default 10,000 RU/s

• This could be improved by adding more RU/s to the FHIR server, thus cost, and it is unclear what performance improvements would look like
• Any parallel operations being done would bottleneck this

Execute Bundle — Users have documented around 120k resources per hour, or ~33 resources/second

*Note: The FHIR loader parallelizes execute bundle calls to “bulk ingest” data. This means instead of a single, long running operation — it is many operations all at once, creating additional cost and throttling the API. (For comparison, Google’s Bulk FHIR Ingest is a free long running operation and 20x faster at default configuration).

Overall

Azure’s FHIR Healthcare API may meet enterprise requirements for size, however, will likely be throttled with streaming or API performance needs. An interesting note for Azure’s offering is the storage and API operations are all dependent on resource units. This means if the resource units are being utilized for say an ingest operation, users must provision additional resources (read: costly) to support their other use cases like read, search, and storage.

AWS

Previous Blog Here.

Scale

The documentation states the maximum is 500,000 resources per database. Based on their documentation of 2 KB per resource, AWS can support 0.001 TBs.

Performance

Unclear product documentation and little documented enterprise usage for there to be reliable threads/community discussions about.

That being said, I do think import performance is irrelevant when the data limits are so low.

Overall

It is unclear if AWS HealthLake could meet either enterprise scale or performance needs.

Unmanaged Services

This option is a traditional way of viewing infrastructure. It requires the customer to provision everything themselves. Most of the information on why to choose managed vs unmanaged is core to Cloud’s positioning and can be found on any number of marketing documentation.

For FHIR specifically there is a lot of work that needs to be done from the server side to horizontally scale the server and database — however, the most difficult task can be creating, managing and optimizing indexes. An enterprise FHIR server that is unmanaged likely requires an additional team of database administrators and infrastructure experts to ensure the solution can meet scale and uptime requirements.

Performance Implications

With unmanaged services, scale and performance are directly related to the database and infrastructure being used. If a SQL server is being used, it will likely not scale to the same size as a NoSQL or GraphQL database. Similarly, performance is going to be correlated with how much infrastructure the server is provisioned on.

For some perspective — IBM recently wrote a blog on their performance testing (way to go, IBM!)

HAPI*

I do not discuss the scale of HAPI in this blog, as IBM out performed HAPI in all three of the performance tests that were conducted.

• HAPI is the backbone of SmileCDR.

SmileCDR

SmileCDR has since published their performance testing and I have included it as an update to this post (1/29/2022). Thank you SMILE CDR for heeding my call, even if you roasted benchmarking in the process :)! I have moved them down to the unmanaged section as well.

Scale

This is the second largest scale test I have seen be published to date (see above). SMILE was tested with 1 million patient records (1 billion resources) which equals about 2.2 TB of FHIR data. I suspect this number was done for time and simplicity of the test, as a user could configure additional infrastructure and the author makes it clear it can be scaled to greater loads.

The maximum capacity of the infrastructure SMILE chose (Amazon’s Aurora PostgreSQL) is 128 TB of data.

** One thing I would love to see from SMILE is how indexes affect the storage numbers. It mentions they turn OFF some default search parameters for customers, as they increase storage space. This is a major point for managed services that can provide full-functionality out of the box without reducing performance or increasing cost and storage.

Performance

Bulk Import — after some digging, it appears that SMILE does not supports bulk import of raw FHIR data. SMILE does support bulk import of CSVs to be transformed to FHIR, as documented here.

Due to this, similar to Azure’s method, SMILE suggests parallelizing the bulk load, in a bundle format.

Execute Bundle — the import performed by execute bundle is impressive at 11,716 resource per second — however this is over 100 upload threads. In order to compare it to other numbers provided above, that averages to about 117 resources per second per thread. That is still the best that has been calculated for execute bundle in this blog, however, per thread is still significantly slower than what one can achieve with a bulk import functionality.

Cost

It wouldn’t be a true enterprise comparison without a cost comparison!

Based on the tests run, SMILE used a 4–6 EC2 instances throughout the test. That has a runtime cost of around $484/month at their maximum use tested.

Amazon’s Runtime: $194.40 ($0.27/hour), Azure’s Runtime: $292.00 ($0.40/hour), Google’s: Free

The storage and use of database costs are on Amazon’s Aurora, and with 2 TB of data and 64 ACUs (their tested max) the storage costs is around $3,000/month. This ACUs model is similar to Azure’s RU/s model where ACUs need to be scaled up by operations.

This is one drawback of not supporting bulk import, where both AWS’ HealthLake and GCP’s FHIR APIs support free import.

All of this is with an asterisk that I do not know the cost of licensing for SMILE and those should be considered as well when doing a true analysis.

Overall

One sticking point, when compared to the other offerings above, is the need to set up infrastructure opposed to simply “click to deploy” models that the Cloud vendors offer. The level of fine tuning someone is willing to do impacts the usability, for example, infrastructure was reconfigured and scaled up/down manually through out the scale test. This can be a positive and negative as I elaborated above, however it does make it more difficult to directly compare to other offerings that do not need infrastructure configuration.

IBM

Scale

The scale of open sourced, unmanaged servers will be tied to what database they are being implemented on. The scale of that database will be the limiting factor to size. The blog does not talk about the size of their implementation tests, however, the authors used PostgreSQL databases. Limits to PostgreSQL databases are not documented, however, in order to meet tens of terabytes of scale users would likely need to shard the system into many servers.

Performance

Performance is tied to how much infrastructure is provisioned. The largest tested infrastructure in the blog was 100-threads on a Kubernetes server, where IBM performed ingest at 750 resources/second.

Overall

An enterprise could likely provision enough infrastructure to meet their size and performance needs, however, it would be difficult to manage and require constant tuning and upkeep.

Honorable Mentions

IBM’s search functionalities were impressive in this exercise, however, even though latency was performant (quick to return) — the authors did not test the scale of many operations at once. With a single server being used as an operational store it should be assumed that it will need to support many requests hitting the API at once and the solution needs to be tested for such.

Limits

This blog covers latency at an extremely high level, however, I cannot account for the networking specifications such as on-premise versus Cloud — and the many networking options enterprise Clouds provide. These limitations should be considered when making a decision regarding FHIR solutions.

This blog also does not get into the specifics of search performance. Search is one of the most important functionalities for a FHIR server to support at scale and users should be requiring search benchmarking from solutions. Specifically, there are documented issues with things like pagination tokens in search (such as with HAPI: https://github.com/hapifhir/hapi-fhir/issues/536) that need to be considered when making a decision for your organization.

Conclusion

Takeaways:

• Almost all servers reviewed — except Google Cloud’s — could likely not handle streaming EHR data into their servers
• Azure should consider supporting bulk FHIR ingest as they are currently operating on an open source functionality that bottlenecks this workflow (and is more $)
• — Google is 20x faster than Azure at default for ingest
• AWS would need to clarify their scale capabilities before performance testing can be considered
• Where is Smile CDR and their documentation?
• Comparing open source/unmanaged systems to managed systems in this context is extremely difficult as they are apples to oranges
• — I applaud IBM for doing benchmarking on their system and releasing the results

Ideally, this blog brings to light the need for clarity, documentation, and performance testing. I hope this encourages organizations to be transparent about their limitations, and thus, the use cases their products are capable of supporting.

The storage and performance implications are the differences in people’s lives. I hope this calls out the questions and expectations decision makers need to be asking of their FHIR solutions.

Author’s Notes

Let me be real for a second — and transparent about my thoughts that I cannot validate with data. FHIR is meant to be an interoperability standard. What this means, beyond just storing and surfacing data, is the connection layer. FHIR is building an ecosystem — it wants to be an ecosystem for connecting data to models, applications, and ultimately patients.

What this really means is the performance requirements I can calculate today (thousands API hits per second) will be exponentially raised going forward. If FHIR becomes what I hope it does, we will need scale and performance far beyond what is outlined here.

Enterprises should be prepared for that future.

Overview

I previously wrote a comparison about Google Cloud’s Healthcare API offering and AWS’s HealthLake offering here. When I did the initial comparison AWS had not released their product for general availability so I did not include the documented limitations for the product. Now that it has been GA-ed, I wanted to add onto the previous publication with usability and scale considerations with AWS HealthLake. The homepage claims to support use cases such as, “manage population health, improve care quality, optimize hospital efficiency.” Upon further evaluation these use cases are not feasible using the out of the box product, as documented in this post.

NOTE: Most of the issues discussed in this blog are based on AWS’ limitations documentation. AWS states that users “can request increases for some quotas, and other quotas cannot be increased.” I attempted to understand which of the HealthLake quotas can be raised by viewing the Service Quotas console, however, none of the HealthLake quotas are listed. I then attempted to file a support ticket, however, HealthLake is not a listed supported product within the service limit dashboard (screenshot below). As far as I can tell, users cannot currently request quota increases to HealthLake. I could not determine what quotas (if any) are able to be raised nor how users could increase quota if available.

Real-World Use Cases with AWS HealthLake

In order to understand the scale issues with HealthLake, let’s start by creating an example health system to base assumptions off of. A typical clinical organization has at least one electronic health record (EHR) system, or often times many, EHRs through out their organization. A common use case systems are implementing is to leverage real-time EHR data to feed machine learning models — shown below.

In total these EHRs can send a few million HL7v2 messages a day. The average number of HL7v2 messages per day is up for debate — OrionHealth quotes 50 million per day here and InterSystems benchmarks at over a billion per day here. In my experience both of those rates are higher than what we see in the wild. Something more practical, for example, is the scale iNTERFACEWARE tests that “closely mimic(s) real world situations” at around 8.3 million inbound messages a day.

Now, let’s assume you’re a small system — your EHR may only produce ~1 million HL7 messages a day (~10/sec). This is a low estimate as it assumes the same rate throughout the day instead of accounting for the ebbs and flows of patient care.

Based on the above, AWS must support ingestion of at least 10 HL7v2 messages per second-for a small hospital system to stream their patient data to HealthLake. Using the defined architecture (above) for streaming EHR data — we need to convert the HL7v2 messages to FHIR resources. A single patient admit message (ADT01) creates 15 FHIR resources based on the open source standards.

This means a small hospital system would create around 150 FHIR resources per second. This is well beyond the AWS HealthLake’s documented limitations for ingestion.

HealthLake’s ResourceCreate operation scales to support 20 resources/sec. Our fictional organization would have to split the streaming pipeline into 8 different FHIR stores to ingest their data — with absolutely no read or deletes allowed.

If our organization chose to use the update operation instead of create, HealthLake’s ResourceUpdate operation scales to support 100 transactions/sec. This would require us to have at least 2 FHIR stores to support our streaming data use case.

You might ask, so what? The issue with multiple FHIR stores is that there is no simple way to aggregate FHIR data from multiple stores. Read: Users cannot easily aggregate patient data if it is stored in multiple FHIR stores. FHIR does not have specs for federated searches and it is difficult to create a single facade over multiple databases as highlighted in this FHIR DevDays talk.

TL;DR — there is no way for an organization to meaningfully use all of their streaming FHIR data in AWS HealthLake today.

So instead, let’s suppose we are only batch loading our data. In Synthea, a realistic synthetic FHIR generator, one patient record is ~546 FHIR resources. AWS’s HealthLake states to only support 500,000 FHIR resources per store. This means each FHIR store only supports 915 patient records.

Therefore, in order to do analytics or searches over more than 900 patients — a user must export the data (for a fee) and send it to a different AWS product. The only supported product integration today is for S3 storage (not an analytics product). Read: analytics requires custom code and pipelines.

HealthLake does not support streaming export, and in fact, the documentation says,

“Only one import or export job can run at time for an active Data Store.” — AWS HealthLake Developer Guide

This means a user cannot do real-time analytics over a HealthLake Data Store. The export capabilities is bottlenecked by a single operation — and that single operation has a limit of one per minute.

Finally — assuming 915 patient records per store, and with a quota of ten stores per account, a single AWS account can only support storage for ~9,150 patients.

A traditional health system that sees around one million patients a year would need a 100x the scale of AWS HealthLake to support their system for a single year.

These product limitations inhibit the very use cases their documentation claims to support. The website states HealthLake can: “manage population health, improve care quality, and optimize hospital efficiency.” Let’s dig.

The first blog on population health states AWS uses the MIMIC-III dataset that has over 40,000 patients. Upon further reading, the actual number of patients ingested into the AWS FHIR Data Store was 1,001 patients (screenshot below) — far less than MIMIC’s 40k patients.

In AWS’ blog for improving quality care using machine learning, the examples also were capped at 1,000 patients (“500 patients with a positive CHF diagnosis…500 patients who didn’t have a CHF diagnosis”).

Summary

I’m hoping AWS provides more support, documentation, or use cases that demonstrate their product in a real world scenario. Based on my research, the current state is not a feasible option for many enterprise customers today.

*This is not discussing the need for reads and searches — most business use cases in FHIR include high-transaction support — such as precision medicine, operational efficiencies, and SMART-on-FHIR applications. Read operations are also throttled to 20 resources per second, even less if users are also running create or delete operations on the store.

Note from the Author

While this is a review of the AWS limits with FHIR, it should not be lost on the reader that there is a bigger theme at play — FHIR at scale. Right now, the community has little solution for supporting use cases that require FHIR data that is stored in different databases. Until we can easily solve this by building a scalable facade or creating a federated search model, users must be considering the scale of their FHIR solution. I hope this article helps users begin to understand the consequences that can come from not appropriately scoping a solution prior to implementation. Just like we do with any other database and server on the market today, I propose we start bringing scalability into the FHIR conversation.

Disclaimer: I am employed at Google Cloud as a Solution Architect.

Introduction

I am a FHIR advocate — and thus, am always interested in the next best addition to the FHIR family. Previously, I wrote a blog on the difference between Azure and Google’s FHIR APIs here. As a continuation of the managed API series, I decided to look into AWS’ HealthLake. AWS announced HealthLake at their Re:Invent conference earlier this year. Unfortunately, it is in preview, so I am unable to test out the product, thus, all of my research is based on their documentation and their conference presentations.

This is an analysis of AWS’ HealthLake product in comparison to Google Cloud’s FHIR and healthcare services. As always, I attempt an unbiased review of both with documentation and resources to support. This reflects the state of both services as of June 2021.

Infrastructure and Scalability

Google Cloud (from previous blog)

The underlying infrastructure for the API is built on Spanner, the same technology that powers Google Cloud Spanner. Google Cloud Spanner is described as a “fully managed relational database with unlimited scale, strong consistency and up to 99.999% availability.” It is a first of its kind database, claiming to break the CAP theorem, having both analytics and transactional functionalities. Spanner ensures “TrueTime” external consistency which is the strongest form of consistency one can achieve for transactions in a distributed system. On top of the Spanner infrastructure, Google Cloud’s Healthcare API is tightly integrated with additional proprietary technologies such as indexing and queuing features, optimized for FHIR workloads, to achieve its performance and scalability.

AWS

Amazon does not have clear documentation on exactly how FHIR data is persisted. Based on their FHIR search functionality documentation AWS states data first is stored in Amazon DynamoDB for CRUD operations and then processed into an Amazon Elasticsearch Service cluster for search operations. It is unclear if that suggests the data is persisted in two separate products based on the users interaction with the product — or if it is removed from DynamoDB once the data has been processed. It is also unclear what the latency/performance implications are by having a data processing pipeline between the two products. In fact, the documentation alludes to some latency as it states the store has “eventual consistency” for searchability.

Comparison

Spanner and DynamoDB have plenty of comparisons done by third parties, so I will not cover the database differences in this article, rather, focus on the difference between the APIs provided by Google Cloud and AWS HealthLake to access the FHIR data. Both APIs leverage performant, high availability back-end systems for their FHIR data storage, yet neither API provides direct access to the underlying datastore.

Architecture

Google Cloud

Google Cloud FHIR service has tight integration with the other offerings in the Google Cloud product stack, as stated below.

BigQuery Integration

First and foremost, there is an out-of-the-box integration with Google’s Cloud data warehouse product, BigQuery. This means with one-click users can bulk export FHIR data to BigQuery, or choose to stream data from the FHIR store to BigQuery real-time. This saves organizations from the hassle of creating and maintaining a custom pipeline between the products, as it is done automatically for users without any code. This also saves users from writing a DDL or maintaining the schema — as Google uses community best practices by leveraging the SQL on FHIR schema.

BigQuery also is a main point of integration for Google Cloud’s ML suite, such as their recently announced Vertex AI. Once data is exported or streamed to BigQuery, users can utilize their data to train and deploy models, enrich with BigQuery’s public datasets, or integrate with any number of partners in the BigQuery ecosystem.

Google Cloud Storage Integration

A second out of the box integration option is with Google’s object storage product, Google Cloud Storage. With the same one-click method used for BigQuery, users can import and export FHIR data to and from Google Cloud Storage. This is useful for data interoperability and sharing use cases.

Google Cloud Pub/Sub Integration

Finally, Google’s Healthcare API has out of the box integration (again, one click, no code) with their Pub/Sub product. This is a real-time notification service for events that occur on the FHIR store. The notification — or message — is published when a resource is created, updated, or deleted from a FHIR store. This notification can be used to kick off any number of real-time workflows, such as triggering an action on an application or to use the new resource as an input for a ML model. Pub/Sub allows for integration with most of the Google Cloud’s data suite.

Overall Architecture

The overall architecture for Google Cloud’s Healthcare APIs is demonstrated in the image below. Note: there are capabilities shown that are out-of-scope for this article.

AWS

HealthLake’s main value add is their integration with Amazon Comprehend Medical, a natural language processing service specific for medical use cases. AWS provides the seamless ability to search on text data that has been through Comprehend Medical and stored in the FHIR format. This is a value add for analyzing free text and clinical notes that may be difficult to abstract otherwise in combination with structured clinical data stored natively in the FHIR format.

Other than Comprehend Medical, the only other data integration Amazon HealthLake has with its product suite is through their object storage product, Amazon S3. Users can import and export data from S3 into Amazon HealthLake in batch/bulk only. The overall architecture for Amazon’s HealthLake is demonstrated in the image below. Note that the only way to utilize Amazon SageMaker, their ML tool suite, is by exporting to object storage, creating a custom pipeline, and pushing it to the rest of the architecture.

Comparison

The main external data integration AWS offers is with their object storage product, S3, for batch import and export. This means Amazon only supports bulk/batch use cases and cannot be utilized for streaming use cases. Google offers the same functionality with their Google Cloud Storage integration, but also has additional integrations with their data warehouse and notification services. Fundamentally, if a user wanted to utilize the FHIR data outside of the CRUD and search operations provided by HealthLake, they would have to build custom pipelines to other products and build their own event management system. Google Cloud has removed the need for users to write or manage any integration code by providing UI-based product integrations and coupling with a comprehensive database notification system.

Accessing the FHIR Data API Comparison

FHIR Servers traditionally offer an API specification defined by the HL7 community on how to interact with the FHIR server. FHIR servers typically need to accommodate for enterprise functionality beyond the API methods defined by the standard FHIR specification, for example, import and export to other systems and a need for analytics on FHIR data. The ability to use SQL directly over FHIR is becoming increasingly popular in the community, as well as the need to visualize the data for dashboards or administrative tasks. FHIR data viewers have also risen in popularity to allow non-technical users access to the raw data without having to manually parse the JSON data to understand the resource. Currently, AWS only offers users to access data in the raw JSON, whether it be in a storage object or from an API return. Google’s suite also offers direct API access to FHIR content as well as integration to utilize the raw JSON as Google Cloud Storage objects. Google has two additional functionalities not supported by AWS’s HealthLake — direct access through a SQL database and a UI to view data. Google allows for structured SQL-querying and viewing of FHIR through BigQuery by a direct integration between the products. In addition, Google has also launched its own FHIR viewer. Google’s FHIR viewer allows users to access their resources through the Cloud Console — providing an easily-digestible way to view FHIR data for all users, regardless of skill set. It provides options for an overview display, a tree-structured visualization (shown below), and a direct view of the raw JSON object. Finally, the FHIR viewer allows users to navigate between referenced resources by simply clicking on the reference. FHIR accessibility is a clear area where Google has increased functionality and usability.

FHIR Version Support Comparison

The following table lists the currently supported versions by each service.

FHIR Operations and Server-Wide Operations Comparison

Import and export functionalities for HeathLake are only designed to be to and from storage buckets, or object storage, not databases. This means that developers do not have query-ability against the data nor can they visualize or analyze the data without creating custom pipelines to other AWS database products. This is in stark comparison with Google’s integration with their managed data warehouse product, BigQuery, which is done automatically for users. There is also no streaming in or out of the AWS product, making it less likely to be utilized for AI/ML workflows. Below is a more in depth comparison of the operations able to be conducted by the two services.

There are also several extended operations the FHIR community has defined that Google Cloud supports that are currently not supported in the AWS product. Patient $everything gives users the ability to fetch all the resources around one patient and is currently only supported in the Google Cloud product. Observation $lastn is a second FHIR wide call that allows users to pull the last “n” number of observations for a certain subject, and is currently also only supported in the Google Cloud product.

Furthermore, Google has robust console integration for their FHIR stores, allowing for import and export to be done on the console UI as demonstrated, however, also creation and deletion of stores — and additional functionality such as deidentification.

Rest API calls Comparison

The table below highlights the differences between the rest API calls of the two services. AWS supports very minimal API functionalities compared to the API specification outlined by the FHIR community.

Search Comparison

Search can be an extremely useful FHIR operation for users. The following table has general search functionalities and what each product supports. On top of this table, Google supports a number of advanced search capabilities such as sorting, chained parameters, and reverse chaining which are not supported in the AWS environment.

Pricing

Note: AWS’ HealthLake is currently free as it is still in preview. This pricing is based on their public documentation and both vendors pricing is subject to change. This pricing analysis was conducted on 05/12/2021.

Google Cloud

Google Cloud’s pricing is based on data storage and API requests. Structured storage has multiple tiers, their free tier up to a GB per month, $0.39 per GB per month up to 1 TB, then it drops to $0.34 per GB per month above 1 TB. The API pricing also has multiple tiers, the free tier covering up to 2.5 billion requests per month. After the free tier the requests are broken down into standard, complex, and multi-operation requests. For standard and multi-operation requests, the API is $0.39 per 100,000 a month, and $0.29 after a quintillion of them (wow!). Complex requests run at $0.69 per 100,000 a month, dropping to $0.59.

AWS

AWS’ pricing is also broken out into data storage and API requests, with an additional runtime charge for server costs. The Data Store cost to run is $0.27 per Data Store per hour. The free tier includes 10 GB of storage, then jumps to $0.25 per GB stored per month. For API requests, AWS offers the first 2.52 million requests per month for free, and then charged $0.015 per 10,000 queries per hour, rounded to the nearest 10,000.

Compare

AWS does have a larger free tier for data storage than Google Cloud at 10 GB versus Google’s 1 GB. That being said, AWS also charges for server runtime, a cost GCP’s FHIR API does not incur. If a user was to run a server 24/7 for 30 days, that would cost around $194.40. AWS has a lower storage cost at $0.25 GB per month, compared to Google Cloud’s $0.39 GB per month. The API cost for AWS is calculated per hour, while Google Cloud’s is calculated per month — making it difficult to make a 1:1 comparison.

I have calculated the amount required to store 100 GB of data for a month while querying the FHIR store 1,000 times a minute for both services.

HealthLake is a cheaper FHIR option for storage and API calls, however, charges almost 200 dollars a month for running a FHIR store. This could be an unfriendly option for smaller organizations or small workloads looking to test FHIR, or, for larger organizations whose strategy revolves around having many FHIR stores.

Note on the API free tier: With 3,500 queries per hour free, this is 56,500 an hour. That being said, since it rounds up to the nearest 10,000 queries per hour, it is still charged at 60,000 an hour — or $0.09 an hour. The free tier for AWS seems to be a little misleading, since they round up to the nearest 10,000. So if a user queried 13,501 times, with a free tier that makes 10,001 queries, yet, AWS would charge a user for 20,000.

Limitations in Preview

AWS has restrictions in preview currently documented here. These limitations are very limiting, and due to these limitations and the early preview status, an enterprise organization would likely be unable to utilize this in production.

Out of Scope

Both Google Cloud and AWS have medical NLP products as part of their healthcare offerings. This article does not dive into the functionalities of either, as the likely most useful comparison would be to review accuracy of each — requiring a third-party study and review.

Summary

Google’s FHIR API when compared 1:1 with AWS’s HealthLake supports double the amount of FHIR resources, all FHIR versions (in comparison with just R4 in AWS) and more capabilities to view and analyze the FHIR data. In a feature by feature review, Google has significantly more API functionalities than what AWS offers — including extended operations and conditional operations. Google’s API has more comprehensive search capabilities and broader support for things like FHIR profile validation and pagination. Google has tighter integration with its larger product suite thus allowing for less overhead and management on users. Google supports streaming of data, unlike AWS’s batch-only model, and thus is a more attractive option for machine learning or application use cases. Finally, AWS pricing is more attractive, however, Google FHIR offers more advanced FHIR capabilities that are necessary for enterprise data management. Try both out and decide for yourself at Google Cloud and AWS.

TL;DR this partnership will allow clinical data coming from EHRs, applications, or other sources in the FHIR format to be utilized with research data in the OMOP standard. This has been a common use case - most notably with GT-FHIR2, the OMOP on FHIR Project. This project was built out of Georgia Tech and allows for mapping between FHIR and OMOP common data model (CDM).

Google Cloud has long recognized the importance of each of these standards and has built out products and solutions for both, such as the Google Cloud Healthcare API, ATLAS on GCP, and provides large, synthetic, public datasets in both the FHIR format and the OMOP data model.

Recognizing the need to interoperate between the standards, Google Cloud also formed a partnership with Odysseus Data Services to assist customers with the conversion of data to OMOP, deployment of research applications such as ATLAS on GCP, and jointly supporting the community with outreach and Google tool integration (like BigQuery support).

Most recently, Google Cloud created an open source healthcare data harmonization tool suite to assist users in translating their data to various clinical standards. We are proud to support the conversion of FHIR to OMOP and look forward to seeing what the new partnership between OHDSI and HL7 will bring to fruition!

For those more interested in the technical aspects of our mapping engine…

Mapping Data to OMOP

The Google Cloud Healthcare team has open sourced a mapping engine that can assist users with harmonizing their data to the OMOP CDM (and many others!). Structural transformation is an important step in building a data transformation pipeline and includes field mapping, parsing, and formatting data. With the open source mapping engine and library, users can easily manage their mapping configurations without the overhead of managing the pipelines.

The healthcare team has also open sourced reference mappings for FHIR to OMOP using the Georgia Tech references to accelerate OMOP adoption for those who already have their data in the FHIR standard. This reference mapping can allow researchers to utilize applications built on the OMOP standard or help further analytics in clinical research use cases.

For more information check out the joint Google Cloud/Odysseus HIMSS session conducted on creating transformation pipelines on Google Cloud for the OHDSI toolsuite!

Using the OHDSI toolsuite on GCP

The OHDSI tool can support many analytics use cases, such as cohort definition, cohort generation and population effect estimation using core OHDSI components (Achilles, WebAPI and ATLAS). These tools have been enhanced to interoperate with BigQuery (and other Google Cloud components), and can be enhanced by use of other Google Cloud offerings, such as Google’s published OMOP synthetic patient dataset and on-going open source efforts to map data to the OMOP standard.

“The introduction of the Google BigQuery into the OHDSI ecosystem allows for unlimited scalability and elasticity for multiple OHDSI collaborators who are utilizing Google Cloud Platform (GCP). The ability to host multi-terabyte data sets while getting fast responses for complex analytical queries enables the execution of internal and collaborative network research. Going forward, Google and Odysseus have partnered in providing ongoing support for OHDSI ATLAS and other tools and methods on GCP platform.”

Gregory Klebanov, Chief Technology Officer (CTO) at Odysseus Data Services

To summarize my thoughts… In someone else’s words

“This partnership closes the loop from research to clinical impact, potentially enabling analytics and models validated on real-world observational data to be put into practice within a health system by leveraging the interoperability standards for healthcare data.”

Marianne Slight, Product Manager for Google Cloud Healthcare and Life Sciences

Disclaimer: I am employed at Google Cloud as a Solution Architect.

Introduction

After spending a week at the FHIR DevDays conference back in June, two main players stuck out as leaders in the fully managed FHIR (Fast Healthcare Interoperability Resources) API Cloud space — Google Cloud and Microsoft Azure. At the event the two organizations had relatively similar demonstrations, showing HL7v2 to FHIR conversions, de-identification functionalities, and visualization capabilities for each. I began to wonder, is there documentation on the difference between the two organizations and offerings? So I decided to dig.

This is an in-depth analysis of the current state for each API. I attempt an unbiased review of both Azure API for FHIR and Google Cloud’s Healthcare FHIR API. This reflects the state of both APIs as of August 2020. Azure launched to public preview on February 7th, 2019 and was generally available October 21st, 2019. Documentation for the product can be found here. Google Cloud launched to Early Access on March 16th, 2018, Beta on April 4th, 2019 and was generally available April 9th, 2020. Documentation for the product can be found here.

Infrastructure

Google Cloud

Although you won’t find it in the public documentation, the underlying infrastructure for the API is built on Spanner, the same technology that powers Google Cloud Spanner. Google Cloud Spanner is described as a “fully managed relational database with unlimited scale, strong consistency and up to 99.999% availability.” It is a first of its kind database, claiming to break the CAP theorem, having both analytics and transactional functionalities. Spanner ensures “TrueTime” external consistency which is the strongest form of consistency one can achieve for transactions in a distributed system. On top of the Spanner infrastructure, Google Cloud’s Healthcare API is tightly integrated with additional proprietary technologies such as indexing and queuing features to achieve its performance and scalability.

Azure

Azure offers two types of deployments for their FHIR API. The first is an open source version that can be deployed on any infrastructure, including Azure, and enables application developers to manage their own server and control the backend. The second is a managed deployment; this article focuses exclusively on the managed API.

For Azure’s managed deployment, the API leverages a managed Cosmos DB instance. Cosmos DB is described as “a fully managed NoSQL database service for modern app development with guaranteed single-digit millisecond response times and 99.999% availability.”

Compare

There have been a lot of comparison articles between Cosmos DB and Spanner, so I will not cover the database differences, rather, the API differences in this article. Both APIs leverage performant, high availability back-end systems for their FHIR data storage, yet neither API provides direct access to the underlying datastore. While Azure has published SLA numbers here, Google has yet to publish their SLAs for the product.

Overall, Google Cloud has less published information on their infrastructure as a fully managed service, which is deliberate since Google has abstracted away the need to understand how the API is built so that users can focus solely on the functionality. A true comparison of the infrastructure would come from the performance numbers and tests against each, which is not covered in this article.

Both API presentations at FHIR DevDays had users leverage something other than their managed service in order to visualize the data. Google Cloud demonstrated the export capability from the FHIR store to their managed Data Warehouse product BigQuery — with no need for schema definition or DDL scripts. They also demonstrated how their recent acquisition of Looker can allow for users to access and visualize the data sans-code. From Azure’s DevDays demo, they leveraged PowerBI by connecting it directly to their FHIR store. While this required more schema definition than the Google counterpart (albeit, UI based) the end result was similar in nature.

Scalability

Google Cloud

Google Cloud’s FHIR API is fully managed and does not require users to configure, set, or otherwise tune their infrastructure in any way. While this may be unfamiliar to legacy or on premise users, this can be a huge advantage for reducing overhead and operational costs usually associated with similar developments or an open source implementation. Not only does the database automatically scale to meet additional capacity needs with no operator intervention, the servers required for operations also automatically scale with use.

Azure

CosmoDB requires users to not only manage their partitioning but also set the number of request units for CosmoDB. Request Units (RUs) are what operators utilize for throughput and/or operations against the database. When the database is provisioned, users set the amount of RU’s they want to reserve for capability — meaning, users have to pay for enough resource units in Cosmos to ensure FHIR resources are available and accessible to users at all times. This may mean users are over-provisioning and underutilizing their infrastructure, or paying for more than what they may need. More about how to configure Cosmos DB can be found here.

Compare

Azure end users have to describe their scaling parameters and configurations in order to meet scaling needs thus it does not seamlessly scale without customer management like its Google Cloud counterpart. Additionally, it may be difficult for users to plan their capacity and usage of resource units, since there is a lack of documentation on how many and what type of API requests require what amount of RUs.

Pricing

Note: This section only touches on the storage pricing of each of the APIs. For API call pricing, the two vendors handle the requests differently (Azure by pre-purchased resource units, Google by on-demand pricing). Due to the lack of a 1:1 comparison, API based pricing is not evaluated. This pricing analysis was conducted on 08/11/2020.

Google Cloud

Google Cloud’s pricing is relatively straightforward. When the API launched to GA, they removed the runtime and server costs. Structured storage has multiple tiers, their free tier up to a GB per month, $0.39 per GB per month up to 1 TB, then it drops to $0.34 per GB per month above 1 TB.

Azure

Azure’s pricing can be broken out into two different categories, server costs and storage costs. Unlike its Google Cloud counterpart, Azure does charge for server cost, which is an additional $292.00 a month if the FHIR server is running 24/7. For storage pricing, Azure’s as-is storage price may look more appealing at first, coming in at $0.25 per GB per month, however, there are additional costs beyond the API storage that users need to consider. According to Cosmo DB’s documentation here, each GB of data also requires 10 RU/s for storage on top of the storage pricing. Since 100 RU/s is 0.008/hour, and assuming users want to store for 24 hours a day for a month (30 days) — that means there is an additional $0.576/GB charge for resource units just to store the data. This turns the $0.25/GB/month that is advertised into $0.826/GB/month (calculation for these assumptions can be found here).

Compare

After calculating the necessary infrastructure costs just to store data, Azure comes in more than double for storage costs of what GCP is (comparing $0.39/GB/month for GCP to $0.826/GB/month for Azure). Google Cloud does not have any runtime costs where Azure has almost 300 dollars a month for each store. When you factor in the fact GCP’s pricing model is pay for what you use, which can be more appealing to buyers, versus Azure’s model is pay for what you think you might need: couple this with Azure’s ambiguity on how many resource units are needed for API calls, various transactions, etc, and the cheaper pricing model GCP has, Google Cloud has a more appealing FHIR pricing model.

I have gone to both pricing calculators and calculated the amount required to store 100 GB of data for a month. Google came in at $155.61 dollars per month (here), while Azure sat at $315.36 dollars per month (here, under “saved estimates”).

FHIR Version Support Comparison

The following table lists the currently supported versions by each API. DSTU2 support may be important to you if your ecosystem includes systems that produce or consume FHIR in this format.

FHIR Operations and Server-Wide Operations Comparison

There are also several extended operations the FHIR community has defined that Google Cloud supports that are currently not supported in the Azure product. Patient $everything gives users the ability to fetch all the resources around one patient and is currently only supported in the Google Cloud product. Observation $lastn is a second FHIR wide call that allows users to pull the last “n” number of observations for a certain subject, and is currently also only supported in the Google Cloud product. Google also has a list of extended operations supported such as concept map translate.

Finally, there is a lack of bulk import functionality in Azure’s FHIR API, as it is missing from their open source documentation and was called out on several community posts. The lack of bulk import for the Azure FHIR store can be difficult for users who are looking to ingest bundles that are already persisted somewhere else. The lack of documentation for their store-wide operations makes it difficult to understand which operations user may not be able to perform. Conversely, Google Cloud supports a wide variety of server wide operations that can be performed on the FHIR store, with supporting documentation here.

Rest API Calls Comparison

Each environment has a set of baseline supported methods based on Google Cloud and Azure’s documentation.

Transaction Bundle Support Comparison

One of the largest advantages Google Cloud has is the support for transaction bundles. Azure’s managed FHIR API does not support transaction bundle types which can make streaming implementations difficult. Transaction bundles allow for multiple operations in a single request — often these transactions are connected. For example, if your bundle has an Encounter and an Observation resource — transaction bundles ensure if one resource fails the bundle fails — versus batch may write one without the other. What this means is that your server might store an Observation without ever recording the Encounter. Furthermore, batch calls do not allow for users to reference other resources inside the bundle, as they do not create the same server-assigned identifiers. This could become detrimental when users are attempting to build something like a Provenance resource in FHIR that needs to be able to reference other resources. This can make it extremely difficult to join resources in analysis that were processed in a batch bundle — as the traditional primary key is missing. For more information on business identifiers, visit: https://www.hl7.org/fhir/resource.html#identifiers

Search Comparison

Search can be an extremely useful FHIR operation for users. The following table has general search functionalities and what each product supports. On top of this table, Google supports a number of advanced search capabilities such as sorting, chained parameters, and reverse chaining which are not supported in the Azure environment.

De-identification (De-ID)

Google Cloud

Google Cloud’s Healthcare API has built-in functionality for de-identification of FHIR resources. Since it is built into the API product, the de-identification functionality is also covered under the Google Cloud BAA and is a true managed service. This also means the API can be called from the Google Cloud Console in the Healthcare Data Browser for added simplicity.

In terms of functionalities, Google Cloud allows users of the de-identification FHIR endpoint to create their own rules for processing FHIR resources and create custom de-identification templates. GCP’s de-identification also supports customization at the field-level and offers additional customization at the infotype (e.g. AGE, LOCATION, etc.) level and sub-field level. On top of the functionalities within the API, there is a second product in Google Cloud, called the Cloud Data Loss Prevention API, that allows for further redaction of strings, images, and tabular data.

Azure

Azure recently released an open source project called FHIR Tools for Anonymization. According to documentation, this tool suite helps users de-identify 17 types of PII both on-premise and in the cloud. The tool is designed to process R4 data and allows for configurations at the field-level.

It is important to understand the workflow of the tool in order to compare it with Google Cloud’s counterpart. The tool pulls data from Azure’s blob storage and processes the data on Data Factory with OSS pipelines. The only pipelines that have been published are bulk pipelines, thus sending bulk FHIR data through Data Factory and back to object storage. The issue lies in the fact there is no bulk-import functionality in the FHIR API as discussed earlier in this post. That means, there is no workflow on how to do data de-identification and to bring it back into the FHIR server in bulk.

Configurations only support the built in methods — keep, redact, date shift, and crypto hash. Azure’s de-identification functionality is also limited, they only support rules-based processing of FHIR R4 data. Azure’s response for free-text sections is to delete them, meaning that the user loses access to a significant amount of medical and patient context within the record. Azure calls out that they do not allow for custom de-identification methods in the documentation here.

Comparison

Azure came later to the table, releasing their functionality in March of 2020 when compared to Google Cloud’s release in 2018. The difference also lies in the fact GCP’s is a managed service that means users are not responsible for the underlying tool set instead of setting up a specific environment for the process like Azure’s. Azure does not have a defined workflow that allows users to store their de-identified data back into the FHIR store once complete, leaving it in object storage. This is a stark contrast to Google’s endpoint which reads from a FHIR store and writes into a secondary FHIR store, with no need for additional products or pipeline.

From a de-identification perspective, Azure’s tools support only structured data and are limited to rules-based processing. They are also limited in only being able to apply rules at a field-level, so there is no logic for sub-field and infotype specifications like Google supports. Finally, Google goes beyond rules-based processing by introducing some of the AI-based processing to detect PII/PHI within sections of free-text. This allows for retaining non-PII/PHI components of free-text fields, which can be useful for downstream analyses and applications.

Google supports DSTU2, STU3, R4 — so the variety of use cases Google Cloud can support is more robust than Azure’s that only supports R4.

Notifications and Database Changes

Google Cloud

Google Healthcare API has the ability to associate every FHIR store with a Pub/Sub topic for push based notifications that can be configured on the UI per store. This is a managed service that automatically scales by distributing loads without pre-purchasing involved. By creating multiple subscriptions to each topic, routing for each notification can be done within the product without processing. To trigger a processing event beyond routing, users have to manually create triggers for Pub/Sub topics.

Azure

Database changes rely on CosmosDB change feeds, which do not currently log deletes or specific types of operations. The change feed can send data to an Azure Function or a Change Feed Processor for further analysis. Users have to manually create triggers for the change feed if utilizing an Azure Function using Visual Studio or CLI tooling, or implement a change feed processor.

Azure’s event hub is designed to store all the change feed data for streaming solutions. There is also no push model supported that is not through AMQP, since the change feed is not push based.

Comparison

The ease of use that Google provides through Pub/Sub when monitoring database changes reduces the amount of custom logic in a streaming pipeline. Pub/Sub seems to be a more packaged solution when it comes to streaming analytics, as it has automatic capturing, storage, and routing built in, unlike its counterpart. The lack of logging for deletion or specific types of operations in the Azure environment, without custom code, could also lead to compliance issues or difficulty in further processing. Finally, Azure lacking support for a push model natively is a drawback when compared with the Google Cloud environment.

Summary

In a feature by feature review, Google appears to have more API functionalities, surrounding data functionalities (such as search and de-identification), and tighter integration with its larger product suite. Based on the storage pricing described in the ‘Pricing’ section above, Google also appears to have a more attractive pricing model, coming in around half the cost per GB (at 100 GB) compared with the Azure API. Try both out and decide for yourself at Google Cloud and Azure.

Addendum

Google Cloud recently open sourced an engine that converts data into a common data model and provided reference mappings for HL7v2 to FHIR R4, HL7v2 to FHIR STU3, and FHIR to OMOP. This push to enable users to bring legacy data and streaming HL7 messages to their FHIR store is an additional differentiator for Google Cloud.