Bringing the Power of Google’s Infrastructure to your Apache Iceberg™ Lakehouse with BigQuery

Speaker 0    00:00:00    
<silence>

Speaker 1    00:00:06    
All right, Victor, how you doing, ma'am?  

Speaker 2    00:00:08    
I'm good. Good. How are you today? Where  

Speaker 1   00:00:10    
Are you calling in from?  

Speaker 2   00:00:12    
I'm calling from Seattle,  

Speaker 1   00:00:13    Seattle. Um, today we'll be talking about bringing the power of Google's structure to, uh, Apache Iceberg. I'm very excited to hear her talk about BigQuery and how all these different pieces fit together. I will be back in 15 minutes and we're gonna have an even longer session for q and a.  

Speaker 2   00:00:35    
My name is Victor and I'm software engineer in BQE. I am working in a group that is, uh, tasked with big lakes and similar technologies, and Apache Iceberg is part of the technologist that we wanted to talk about today. So I'm going to go over the general overview of BQE and then I'm going to discuss several existing integrations that we have, uh, that work with, uh, Apache Iceberg. A few years back, we didn't have anything like four years back, and now we have several offerings and we're hoping to have even more soon in general, BQE is, unless you Dunno, is a fully managed and serverless system for data processing. It has completely decoupled storage and metadata layers which are scaling independently and on demand. And basically users and customers don't need to think about scaling or especially having their resources running while there are no jobs or no cures to execute.  

Speaker 2    00:01:51    
So this is different from than traditional node-based computing warehousing, where you have to have your nodes possibly running at all times, even if there are no serving any workloads. So this gives you, uh, big flexibility and cost control over what you have to do. Uh, the engine is known as dremmel, also scales extremely well horizontally and vertically, and provides, uh, uh, modern SQL 2011 compile land SQL language and CLI rest API and other drivers that you can use to talk to b curing. Now the gamo of tables that B curri provide is pretty wide. So on the left there are self-managed Iceberg tables or even self-managed external tables. There, the data is read only. You manage all the data, you manage all the metadata and PQ just provides you curing. On the other side, you have the BQ manage tables, which leverage all the, uh, capabilities that we have.  

Speaker 2   00:03:06    
The storage is managed, the metadata is managed, it's all far and forget, and it gives you all the flexibility and control over the data and metadata on the other side. Because this flexibility and performance come at a cost that, uh, both the data and metadata are in proprietary format. And unless you export or uh, take out your data, you don't have direct access to the data except the, uh, methods that BQE provides be that qri engine or the Read API. But it provides all the features that enterprises require, automating, scaling, depending on the loads, uh, controls and features, compliance, disaster recovery data, residency requirements and column and role level security and whatnot, including materialized views and so on. And in the middle we have our more recent offering, which is Apache, uh, <inaudible> tables for Apache Iceberg, which provide you, uh, some middle ground. The metadata is still managed by B qi, but the data is in open source parquet, and you can export, uh, metadata in, uh, Iceberg format to QI from external <inaudible>.  

Speaker 2   00:04:24    
We'll get to those details later. So the self starting on the left is the self-managed, uh, of Iceberg tables. They come in two flavors and we'll go over the capabilities of each one. So one are external Iceberg tables. In that case, users basically have to manage the data and metadata themselves. It can be done the spark or rubber engines. And Google cur basically receives the pointer to the metadata. And at that point it executes the standard curing, uh, Apache Iceberg curing, uh, processes, reads the manifest files, reads the parque files, and serves the QI by virtue being part of the BQ execution engine that you are gaining. On the other side, all the controls that you might not have otherwise in the open source such as uh, row and column level security, uh, materialized use and search indices. Uh, so in that case, for example, you can generate your data stream it using your Flink into a big lake that will be managed as an Apache table, but later your data can curate the data by applying those positive security and visibility controls with data masking and columnal security.  

Speaker 2    00:05:58    
Uh, the next, uh, level up is the BQ meta store, which is our most recent offering that, uh, recently has been, uh, recently graduated to general availability. This executes as a standard Iceberg plugin where there is a JAR file, or now it's currently merged into, uh, upstream Iceberg repository. So it will be available in every release starting one 10. At that point, the code or the library on the engine site will manage all the file interactions, the metadata and whatnot, but the BQE will act as the ultimate catalog and the source of truth for the metadata. It'll store the, uh, table information, the pointer to your metadata and will ensure that all the data is correct and it leverages the same, uh, availability. Disaster recovery features for that part, that b Curious stores, uh, in addition, if you curate those tables, the BQ rather than through your open source engine, you are receiving same exact, uh, feature as we talked just now for externalized work tables, including columnal security and Possibility building of materialized use and search indices over your data.  

Speaker 2    00:07:24    
Uh, in addition, because by virtue of it being a library, you can still use it directly from, uh, your open source engines, but it is also possible to run it from, uh, other GCP offerings such as Dataproc, uh, which natively integrates with it and just works out of the box as your another meta store. And as I just said, yeah, it leverages all the same curing engine, which is ramel and features all the enterprise features that might be required. Uh, now going to the managed Iceberg tables, which, uh, will be soon launching to general availability. Uh, in this case, BQE is the source of truth for both metadata and the data. And the data is stored in the per K format in the user GCS bucket. And except for one case we'll discuss later immediately at the completion of your DML operations or load operations, it will be available in the bucket. The metadata currently is expertise on demand. So the metadata is stored internally in our proprietary formats, but the Iceberg metadata can be exported on demand whenever user needs that, and soon it'll be coming to, then it'll be out automatically exporting it, uh, as the data inside the table changes. So you will have, uh, fresh Iceberg metadata available in your, uh, GCS bucket for consumption by external engines almost immediately.  

Speaker 2    00:09:14    
Uh, because the data of the, because the metadata is stored in the our proprietary formats, it allows same optimizations that, uh, we can apply to our native tables, which means that there is, uh, there are features like, uh, enterprise control as we discussed, uh, the column row and lower level security data residency and whatnot. Uh, in addition, it supports several more advanced features. One is high throughput streaming using right API, which is one of the best offerings, uh, in the market. It's allows you to stream several gigabytes of data across all your streams per second. And, uh, here the difference from the, uh, previous slide is that that data will be stored temporarily in, uh, internally while it's being converted into per K format. And we'll go over architecture in the next slide. Uh, another thing that, uh, BQE tables for Apache provide is that because we own all the metadata internally, we are able to apply same data optimization, ensuring the PERQUE files are sized appropriately for optimal QE performance. We're applying our internal appropriate algorithms to determining the sizes, uh, computing statistics to ensure the proper performance and whatnot, data re clustering and, uh, garbage collection. So as soon as the data is going outside of your time travel window for curing, uh, the data is collected. So you don't have to, uh, pay your GCS bill unless you need the data for longer.  

Speaker 2    00:11:16    
So going back to the hyper streaming, I wanted to discuss a little bit the architecture that powers it. So the data is arrives at, uh, a right API front end, which is supported by several lan, uh, several libraries in various languages. At that point, the data can be partitioned to several stream. It can be sent by one stream, and it's all being consumed by our stream servers in the backend. Uh, those stream servers collect all the data and store it in our internal proprietary formats for the time being. While the amount of data, which is expected amounts that would necessitate its conversion and export into per k format and the metadata, uh, when that time arrives, uh, the internal system will convert all the temporary data, convert it into per k format and export into your bucket, and the metadata will be merged into, uh, the rest of the table metadata, and then it can be exported and processed by external engines, uh, should the need arise. Interestingly though, if you want to access the temporary stored data right now, you will need to use either BQE engine or the read API connector from external open source engines. Uh, while the data is stored, obviously it's also subject to all the controls as discussed above, including, uh, cec if your table is encrypted, if your custom key, then it will be encrypted and all the data will be stored from BQE. You can always, uh, curate all the data, the union of the recently streamed and, uh, already committed data to the table.  

Speaker 2   00:13:16    
Uh, so the features that are coming, uh, soon with the BQ managed tables are, it's going to be general available soon, and it it'll include continuous Berg snapshots. It'll include time travel support for both, uh, from BQE and from the USS engines. And it'll have, uh, change data capture capability in the right API and high throughput streaming where you not only can write data as pen only, but also you can send changes such as deletions or asserts, uh, which is a highly requested feature. And, uh, as it has been announced that the next, uh, we are also working to support, uh, Iceberg arrest interface so that it is possible to manage the Iceberg tables in your que the arrest catalog rather than the, uh, plugins jars or the code running on your cluster side. Uh, and that's what I had planned for you today, and, uh, I'm ready to answer any questions you have.  

Speaker 1    00:14:30  
Awesome, Victor, thank you very much, uh, for coming and sharing with us. Can you share your screen one more time? And I, and can you go back to that system diagram

Speaker 2    00:14:41    
Uh, certainly.  

Speaker 1   00:14:45    
Okay. By the way, I'm All right. So I'm this one, I'll put my question aside for a minute. I'm already seeing some questions then, uh, show up. So first is, what is the speed of high throughput streaming?  

Speaker 2   00:14:57    
So, as I said, like the ingestion capabilities are above, uh, several gigabytes per second across all streams. Like if you're streaming only stream, it's a bit restricted. Uh, but if you have several streams and you have several streams suggesting yeah, it's, the default limit is three gigabytes, but if you have a good use case and uh, it can be increased even above that.  

Speaker 1   00:15:25    
Yeah. Um, okay. John is asking, is there any cost to the customer for the internal temporary storage and the conversion to Parquet?  

Speaker 2    00:15:34    
Uh, no. So the bill is basically you pay for the right API, whatever the price is. It's standard as for native tables. As for Apache tables, there is no additional central line billings for, uh, either temporary storage or conversion.  

Speaker 1    00:15:54    
Okay. We got another one from VIN note. Are you able to comment on how the table types are distributed across BigQuery native format versus Iceberg or open table formats?  

Speaker 2    00:16:06    
Uh, I don't think I have, like you mean like the number of tables or like proportions of tables of native to Weisberg, I assume. I don't think I have that information, but currently the majority of tables that B Curri has are native tables.  

Speaker 1    00:16:23    
Yeah. Yeah, I think, yeah, he clarifies, you know, the percent of tables in each category.  

Speaker 2    00:16:27    
Uh, no, unfortunately I do not have that information.  

Speaker 1    00:16:31    
We've got another one here. And does BigQuery differ on compute slots? So limited to 2000 per GCP project when querying self-managed versus native versus big lake tables.  

Speaker 2    00:16:46    
So the slots are basically shared across the compute engine, like, so the dremmel is the execution engine and whatever curious are done through dremmel are built to your slots here, regardless of the table type.  

Speaker 1   00:17:04    
Let's see. We have another one here. Uh, surrender is asking Victor, can you explain about time travel in BigQuery Iceberg,  

Speaker 2    00:17:11    
Uh, in the Iceberg? So right now it was, uh, because the product was, uh, in the preview phase, there were some edge cases. We didn't want to release it to general availability for people to use if we weren't sure of the quality at the time. So those are all fixed and will be released. And it works standard way. So if we're talking about internal metadata, it's supported out of the box is for Iceberg, like for time travel, you need to store appropriate snapshots and snapshot history in your metadata, and that will also be available so you can curate with time, uh, travel from your open source engines as well.  

Speaker 1   00:17:55    
We got another one here from Taz. Does your system only support Parquet format or also supports other formats like ORC and <inaudible>?  

Speaker 2   00:18:05    
So depending on which one. So external Iceberg Table supports both C and ro, the managed AP Berg support only per kt.  

Speaker 1    00:18:22    
Okay. Let's see. I think, by the way, folks, you could, uh, feel free to just, uh, drop your questions in the q and A tab and then others can upload them. Um, otherwise I kind of have to fish them from, from the chat and it's a little more, at least I, I don't then know which ones are most popular. We got another one from the chat. Can you comment on how a BigQuery costs, uh, on slowly changing dimensions operations,  

Speaker 3   00:18:49    
Uh,  

Speaker 1    00:18:51    
Slowly changing dimensions operations? Um,  

Speaker 2   00:18:54    
Yeah, I'm not sure I understand the question.  

Speaker 1   00:19:01    
Yeah, I don't know if, do you guys follow  

Speaker 2    00:19:03    
Up?  

Speaker 1    00:19:05    
Yeah,  

Speaker 2   00:19:07    
Uh, yeah, I'm not sure I understand the question well, sorry, but, um,  

Speaker 1   00:19:16    
<inaudible>, if you wanna explain a little bit more, what do you mean by costs? Do you just mean like the extent to which slowly changing dimensions are, um, like operations on, on slowly changing dimensions or like, like adding up to increasing costs?  

Speaker 2   00:19:35    
Yeah. In general for manage the storage is basically built on how much storage you have for the Iceberg tables. Obviously whatever GCS is billing is whatever, uh, how much data you are putting into your bucket. Yeah, so slow changing or fast changing, it doesn't, uh, affect much.  

Speaker 1    00:19:58    
What about any plans for managed Delta leg tables?  

Speaker 2    00:20:03    
Uh, no, there are currently none that I'm aware about.  

Speaker 1   00:20:09    
Okay. We're almost  

Speaker 2    00:20:10    
Colleagues are those supported same way as Berg tables, the, uh, self-manage. So you are able to curate those through B Curri, but, uh, they are readly,  

Speaker 1   00:20:20    
We, um, okay, we got another one here. Is it efficient to use native BigQuery tables versus creating external tables in BigQuery with data and blob storage written in, in, uh, in Huie or Iceberg?  

Speaker 2    00:20:36    
So we, yeah, especially if you're going to curate them from BigQuery, it's more efficient to create them ambi because we have all the necessary metadata internally, which is faster to load than going to the GCS bucket and loading it from there. And the format is, uh, internal and property, which is faster for us to read than it would be RO or Jason in general. So.  

Speaker 1    00:21:04    
Okay. I think we're gonna keep it to the last one for this session. John Esperanza is asking, is there any configuration available for data compaction, uh, on Iceberg tables

Speaker 2    00:21:15    
So those, so for managed tables, I assume so those are all managed internally, but I are algorithms, they're changing over time. They're not, uh, really public, all the thresholds and whatnot. Uh, based on our history and uh, Curie performance, we determine what would be the optimal sizes for our engine, and that's how we determine those.  

Speaker 1    00:21:44    
Yeah, that's fair. Especially for the managed solutions. Victor, thank you very much for coming and sharing with us and joining us for this session.