Operating large data pipelines requires an orchestration layer that scales smoothly as workloads expand. When your pipelines process millions of complex data points every day to feed predictive models, staying up-to-date with your technology stack is a strategic necessity.

Pine59 provides location intelligence data through data pipelines that produce analytical metrics on cadences ranging from hourly to quarterly. One of the company’s most data-intensive metrics, Daily Foot Traffic, computes data for as many as 14 million distinct locations in a single job. To handle this massive volume, Pine59’s system runs entirely on Google Cloud, with the heavy lifting in BigQuery and all of it orchestrated by Managed Service for Apache Airflow (formerly Cloud Composer) running Apache Airflow 3.

As the company’s volume of data and number of machine learning workloads scaled up, Pine59 decided to modernize its monorepo, which contains hundreds of directed acyclic graphs (DAGs). Here is a look at how that transition improved Pine59’s MLOps capabilities, developer workflow, and pipeline speed.

Proactive modernization for growth

Pine59 has long relied on a shared monorepo with code and tooling spanning multiple projects to run its metric production pipelines. As it considered its infrastructure’s future, the company wanted to help its data pipelines run faster and more reliably.

That’s why it decided to stress-test production workloads against the newly available Managed Airflow (Gen 3) architecture running Airflow 3. The initial results were unambiguous: the Gen 3 environment delivered immediate and significant processing speed, task scheduling, and overall stability improvements. Recognizing the clear potential for performance gains, Pine59 initiated a full transition to the new environment.

Orchestrating advanced MLOps

Pine59’s pipelines don’t just move data; they drive complex ML models, so a core aspect of its migration was optimizing the orchestration of its ML inference workloads.

Previously, Pine59 had used standard Kubernetes operators for these tasks. By moving to Managed Airflow (Gen 3), which features a highly optimized and abstracted infrastructure layer, the company’s engineering team refined its MLOps architecture. They did so by setting up a dedicated Google Kubernetes Engine (GKE) cluster that was specifically optimized for model inference and integrated it into the Pine59 pipelines.

This clear separation of orchestration and heavy ML execution compute allows data processing and model inference to run efficiently, showcasing Managed Airflow as a resilient, scalable backbone for enterprise MLOps.

Supporting developers with custom extensibility

Beyond infrastructure improvements, Pine59 was also able to immediately capitalize on Airflow 3’s delivery of a vastly improved developer workflow and user interface. Indeed, managing hundreds of interconnected DAGs requires excellent observability, and Pine59 found Airflow 3’s plugin authoring system remarkably easy to use.

To improve internal developer velocity, the company quickly built a number of custom plugins that it integrated directly into its new Airflow UI:

• BigQuery Auto-linkify: A tool that automatically detects internal BigQuery table references within the Airflow Logs and XCom tabs, dynamically generating direct links to BigQuery Studio for faster debugging (available as a public GitHub gist)

• DAG Run Configuration Search: A custom search form added directly to the DAG overview page. It allows Pine59 engineers to query specific key-value pairs within DAG run payloads (configs) and instantly surface matching runs. This in turn drastically reduces troubleshooting time.

In addition, the team also deployed a compatibility shim layer within its monorepo. This “compat” module dynamically abstracts logic between Airflow versions, streamlining operator migration across versions.

Faster, more reliable pipelines

For Pine59, migrating to Managed Airflow (Gen 3) with Airflow 3 has yielded clear, quantifiable results.

The most important improvement was the speed of its DAG runs. In the company’s previous setup, tasks often got stuck in a queued state during peak processing surges. With Gen 3, queue latency has dropped dramatically, allowing tasks to start running almost immediately.

Consider the comparison below of total aggregated “queued” & “running” time of more than 300 runs of the same DAG between Managed Airflow (Gen2) with Airflow 2.11 vs. Managed Airflow (Gen3) with Airflow 3.1 below. As we can readily see, the difference in queued time is significant.

Coupled with internal DAG optimizations made during the transition, the performance gains are also highly tangible. For example, the Daily Foot Traffic pipeline previously took nearly 38 minutes to complete. With the new instance, the same workload now takes less than 26 minutes —nearly 32% less processing time.

Today, Pine59 processes all its production workloads on its new Managed Airflow (Gen 3) instance. By moving to this next generation orchestration, the company improved its MLOps capabilities, equipped its developers with better tools, and built a faster, more resilient foundation for future workloads.

If your engineering team spends more time managing infrastructure than delivering value, consider a similar transition and discover how it can help you move from maintaining servers to building the future of your data and AI pipelines today.

^{Special thanks to the following contributor to this post: Alexandre Crespo-Perez}

Apache Spark is now the de facto standard for data engineering, data exploration and machine learning. Just likeKubernetes (k8s), is for automating containerized application deployment, scaling, and management. The open source ecosystem is now converging towards utilizing k8s as the compute platform in addition to YARN. 

Today, we are announcing the general availability of Dataproc on Google Kubernetes Engine (GKE), enabling you to leverage k8s to manage and optimize your compute platforms. You can now create a Dataproc cluster and submit Spark jobs on a self-managed GKE cluster. 

Dataproc on GKE for Spark (GA)

K8s builds on 15 years of running Google's containerized workloads and the critical contributions from the open source community. Inspired by Google’s internal cluster management system, Borg, K8s makes everything associated with deploying and managing your application easier. With the widespread adoption of k8s, many customers are now standardizing on k8s for their compute platform management. We are observing a trend towards building applications as containers, to simplify application management among the many other benefits such as, improved agility, security, portability. 

Dataproc on GKE, now in GA, allows you to run Spark workloads on a self-managed GKE cluster. Letting you derive the benefits of fully automated, most scalable and cost optimized K8s service in the market.You bring your GKE cluster and create a Dataproc ‘virtual’ cluster on it . You can then submit jobs and monitor them the same as you would for Dataproc on Google Compute Engine (GCE). You use the Dataproc Jobs API to submit jobs on the cluster, you cannot use the open source Spark Submit directly. Jobs on Dataproc are submitted as native containers and you even have the ability to customize the containers to include additional libraries and data for your applications. 

Concepts for existing Dataproc on GCE users

Node Pool Roles
Dataproc uses GKE node pools to manage the Dataproc cluster configuration. You have the ability to select the machine type for the node pools. All the nodes in the node pool use the same configuration. Configuring the node pool with the following roles allows you to optimize Dataproc cluster configuration. 

• Default: You must have at least a default role for a node pool. If other roles are not defined, default is used to run the workload. 

• Controller: If defined, Dataproc control plane runs this node pool. This role has very low resource requirements. 

• Spark Driver: if defined, Spark job drivers run in this node pool. This allows you to optimize the cluster configuration to the workload characteristics.

• Spark Executor: If defined, Spark job executors run in this node pool. This allows you to optimize the job executor environment. 

Workload Identity
Dataproc uses GKE Workload Identity to allow pods within the GKE cluster to act with the authority of a linked Google Service Account. This is very similar to the default Service Account for the Dataproc on GCE.

Autoscaling
Dataproc on GKE utilizes the GKE Cluster autoscaler. Once Dataproc has created the node pools, you can define autoscaling policies for the node pool to optimize your environment.

Key Benefits

Preview customers with expertise in GKE were able to easily integrate Dataproc into their environments and are now looking forward to migrating Spark workloads and optimizing their execution environments to improve efficiency and save costs. Our advanced customers are exploring GPUs for improved job performance to meet their stringent SLAs needs. As we go GA, these customers are excited about utilizing the advanced k8s compute management and resource sharing for their production workloads. 

Running on GKE enables you to take advantage of the advanced capabilities of k8s enabling you optimize costs and performance by running: 

• Completely independent jobs on the same cluster.

• You can now share a Dataproc cluster among multiple applications with distinct libraries and dependencies. Each Job can run its own container. Allowing independent Jobs with conflicting dependencies to run at the same time on the same Cluster. Earlier, each job with a distinct environment required an exclusive cluster. Relaxing this constraint enables customers to further optimize their execution environment. 

• Multiple clusters on the same node pool.

• You can share the same infrastructure across multiple Dataproc clusters. You can run multiple Dataproc clusters on the same node pools, thereby allowing you further optimize costs. Some customers are now sharing multiple development environments on the same infrastructure. The same is applicable for testing, validation and certification environments. 

• Multiple Spark versions on the same infrastructure

• You can easily migrate from one version of Spark to another with the support for multiple versions on the same node pool. Your cluster management is simplified as you do not need to create two distinct environments and do not have to plan scaling down ‘existing’ cluster and scaling up the ‘upgraded’ cluster. 

Dataproc on GKE Key Features

Following are some of the salient features of Dataproc on GKE

• Spark Versions: You can run Spark 2.4 and Spark 3.1 jobs on Dataproc on GKE clusters.

• Metastore Integration: You can integrate Dataproc on GKE with Dataproc Metastore.

• Job level access controls: You can now specify granular access controls at job level leveraging the k8s RBAC and workload identity. 

• Uniform Dataproc APIs: you can use the same Dataproc APIs to manage clusters, submit jobs and use the same monitoring capabilities as Dataproc on GCE.

Running Spark jobs with the infrastructure management style of your choice

With the general availability of Dataproc on GKE, organizations can now run Spark jobs on their infrastructure management style of choice: Serverless Spark for no-ops deployment, customers standardizing on k8s for infrastructure management can run Spark on GKE to improve resource utilization and simplify infrastructure management. Customers looking for VM-style infrastructure management can run Spark on GCE.

What’s Next

We are actively working on integrating Dataproc on GKE with Vertex AI Workbench for data scientists in the upcoming months. With this integration, data scientists can use notebooks for their interactive workloads and even schedule notebooks executions. We are also looking to extend Enhanced Flex Mode Support to Dataproc on GKE allowing you to maximize the benefit of preemptible VMs.To get started, check out this quickstart link.

You can now take your knowledge of k8s compute management and leverage Dataproc on GKE to run Spark workload.

Today, we’re announcing the Database Migration Program, a new and stress-free approach to migrating existing open source and proprietary databases, whether on-premises or in the cloud, to Google Cloud’s industry-leading, managed database services. With the Database Migration Program, you benefit from assessments, tooling, best practices, and resources from our network of specialized database technology partners. The program also offers special incentive funding to offset migration costs, helping you to quickly and cost-effectively migrate your databases to Google Cloud. Get started today with the Database Migration Program.

Over the past decade, companies big and small have realized the benefits of the cloud for their application modernization journey, helping them become more efficient, scalable, agile, and innovative. Furthermore, managed cloud databases typically result in an overall lower cost of ownership while upskilling database administrators to focus on higher-value work like data modeling and deriving additional value from data with AI and machine learning.

Still, many companies continue to self-manage databases on cloud instances or leave databases on premises even when the application is running in the cloud. The primary reason is the complexity of database migrations. Databases are at the core of every enterprise’s day-to-day operations, making them more challenging to move without careful planning and execution. In addition, migrations can be expensive, time-consuming, and risky. Timelines can drag on and scope regularly increases, leaving customers frustrated. 

Our new Database Migration Program seeks to address database migration complexity by providing comprehensive guidance and support for your migrations. Our assessments help you understand the footprint of your database fleet, its dependencies and architecture, and our specialized database partners can help with their expert knowledge of tooling and resources to move data and code without disrupting your business. Additionally, Google Cloud offers special incentive funding to offset migration costs, helping you to quickly and cost-effectively migrate your databases with the minimum amount of financial risk.

The secret to stress-free database migrations

What’s unique about this program is that you have access to a one-stop shop for all things database migrations. You can break your migrations into smaller sprints and execute one migration after another, allowing you to achieve business outcomes faster. With the Database Migration Program, you can accelerate your move from on-premises, other clouds, or self-managed databases over to Cloud SQL, Cloud Spanner, Memorystore, Firestore, and Cloud Bigtable.

Already, the Database Migration Program is transforming the way our customers and partners approach their database migrations to the cloud, allowing them to reimagine the time and resources required to deliver on their digital transformation goals without the burden of uncertain timelines and high costs.

Three steps for a successful database migration

The Database Migration Program guides you from the initial assessment and planning phase to eventual migration with the expert assistance of qualified database partners. 

Here’s how Google Cloud helps at each stage of the database migration journey: 

1. Assess: Request a database assessment to discover and analyze your existing databases and applications. Leverage specialized tools and resources, along with assistance from Google Cloud database experts who provide guidance based on your specific needs and requirements.

2. Plan: Connect with specialized database partners who can help you create a migration plan, including engineering resources and cost estimates, and identify the right workload to kick off your migration. 

3. Execute: Get special incentive funding to offset some of your migration costs by helping to pay for the specialist technology partner who performs your migration. There’s no need to move everything over at once—you can move one department or database at a time and use the program again as many times as you need.

Interested in learning more? Complete this form to get started.

At Google Cloud, we strive to offer a great customer experience for enterprises by building a robust and supported platform for running all Linux-based workloads.

This mission is why we were one of the first cloud providers to offer purpose-built Rocky Linux images when Rocky Linux debuted last year as a replacement option for CentOS. We were also one of the first hyperscalers to sponsor the Rocky Enterprise Software Foundation (RESF) to support the open source community behind this Linux distribution. With these efforts, we’re pleased that many customers are already running Rocky Linux in Google Cloud today.

Today, we’re excited to announce that we’re taking another step in furthering the support we provide for Rocky Linux. We’re partnering with CIQ—the company started by CentOS co-founder and Rocky Linux founder Gregory Kurtzer featuring core expertise across Linux, cloud, HPC, containers and security— so we can provide customers a new and improved experience for Rocky Linux on Google Cloud. 

Starting today, customers can leverage Google’s support offerings to file support cases for Rocky Linux. Google support teams and the Rocky Linux experts at CIQ are working together to address customer issues to help ensure they get enterprise-grade support. If you already have a paid support plan with Google, you will be able to open a case for an issue related to Rocky Linux. Google teams can expediently help resolve the issues, backed by CIQ expertise, giving you an integrated experience of using Rocky Linux on Google Cloud. 

"We asked ourselves, how do we bring the best value to everyone? Through this partnership, anytime you use our Rocky Linux on Google Cloud, both Google and CIQ jointly have your back! From the cloud platform itself, all the way through the enterprise operating system, every aspect of using Google Cloud is supported by a single call to Google, and together, we are your escalation team.”—Gregory Kurtzer, CEO of CIQ and Founder/Director of Rocky Linux and the RESF

In addition to CIQ-backed support for Rocky Linux, Google is also working with CIQ to provide a streamlined product experience - with plans to include performance-tuned Rocky Linux images, out-of-the-box support for specialized Google infrastructure, tools to help support easy migration, and more. We’re doing these updates in a community-friendly way. Together with CIQ, Google is helping to create a Rocky Linux Cloud SIG that aims to provide optimized, standardized, and simplified Rocky Linux experience. 

If you’re currently looking for alternatives to CentOS as it reaches end of life, Rocky Linux on Google Cloud can have you covered both from a product and support perspective. So, take Rocky for a spin if you haven’t already, and if you have questions or suggestions on how we can help you, please don’t hesitate to reach out to us. To learn more, please also join us for a webinar discussion on April 6th 2022 at 11.00am PT.

Technology leaders are under mounting pressure to modernize infrastructure, control multi-cloud operational spend, and build data foundations for generative AI. However, the discovery required for that level of transformation can entail weeks of manual spreadsheet analysis, mapping in-house infrastructure, and reconciling siloed, piecemeal cost estimates across disparate teams and sources. To help, we’re announcing AI-powered Quick Assessments in Migration Center, which delivers near-instant total cost of ownership (TCO) modeling and automated service mapping.

Compare this to legacy assessment processes, which can stall digital transformation initiatives before they even launch. Manual discovery can delay migration timelines by months, increase engineering overhead, and often miscalculates complex financial models. By replacing manual discovery with AI-assisted automation, IT gains instant, actionable visibility into the TCO and return on investment (ROI) for a given migration initiative. 

Now, organizations can generate comprehensive migration financial models in minutes rather than months. Teams ingest raw infrastructure data or cloud billing reports and quickly receive an optimized target bill of materials (BOM), service mapping coverage, and projected savings. Decision makers interact with an agentic assistant that explains underlying financial assumptions, recommends technical cost optimizations, and exports ready-to-share executive reports.

Inside the AI-powered Migration Center

This is made possible with AI-assisted Quick Assessments alongside enhanced Cloud Billing assessment capabilities, both integrated into the new AI-powered Migration Center.

AI-assisted Quick Assessment for on-premises workloads

Designed for enterprise customers and partners, AI-assisted Quick Assessment automates on-premises infrastructure evaluation to provide rapid financial modeling. Let’s walk through these new capabilities: 

• Instant Compute Engine TCO estimates convert VMware inventory exports (such as RVTools) or aggregated infrastructure inputs into precise Compute Engine cost targets:

• Advanced architecture modeling supports latest-generation Gen4 compute instances and high-performance Hyperdisk storage pools:

• Customizable financial controls allow teams to adjust on-premises baseline cost assumptions to match internal accounting standards:

• Context-aware agentic chat recommends tailored technical cost optimizations aligned with your specific business constraints (such as regional location or compliance needs), clearly explaining the underlying logic and financial assumptions.

• Automated Business Case and Google Sheets export generates ready-to-use reports capturing the complete recommended BOM, TCO comparison, and ROI analysis:

The path forward

Modernizing your infrastructure starts with fast and accurate data. Migration Center’s Gemini-powered features simplify cloud evaluation, empowering IT decision makers to build defensible business cases generated by machine-learning.

Try Migration Center directly in the console today, or take a free migration and modernization assessment to evaluate your workloads and accelerate your strategic cloud journey with Google Cloud.

For too long, enterprises with legacy mainframe estates have been faced with a high-stakes dilemma: continue maintaining their mainframes, essentially kicking the modernization can down the road (fully aware that delayed action only compounds future technical debt), or perform "big bang" modernization with more unknowns and risks.

At Google Cloud, we propose an alternative: a modernization strategy that leverages the power of AI, agility of the cloud and allows for iterative and continuous modernization. This approach recognizes a fundamental truth: mainframe modernization isn’t a pure code-to-code conversion problem. Sure, modernizing a single, isolated and small application is relatively easy, especially with recent advancements with AI and large language models. The real challenge lies in modernizing at real-world scale without breaking the intricate web of dependencies and legacy data formats you find in a large global enterprise, all while ensuring functional equivalence.

For example, some of these real-world challenges include:

1. Tightly coupled data and logic. Application logic is fused directly to legacy, proprietary databases and record schemas, requiring both data modernization from legacy formats and application code rewrites to go hand-in-hand.
2. Integrated transaction monitors where a single transaction scenario can span across millions of lines of code, proprietary mainframe utility suites, and internal/external boundaries using legacy protocols.
3. Undefined application boundaries. Complex inter-dependencies create a tangled "spaghetti code" environment. Resolving this requires decomposing the monolith to isolate migratable units and map them to individual business processes.
4. Intricate sequential workflows, with highly complex execution paths that rely on dense, conditional step logic and rigid sequence dependencies.

In other words, real-world modernization of mainframe applications is so much more than converting COBOL to Java. You also need to modernize the underlying data models, handle decades of obscured application dependencies and interfaces and modernize the underlying data stores. Most importantly, you need to validate and de-risk the modern code with actual production traffic before going live.

Our approach combines the advanced reasoning and scale of our Gemini models for code understanding, with mainframe-specific modernization products to address real-world complexity and challenges. Our solutions span four core pillars: assessment, modernization, de-risking, and data migration.

Let’s take a look at each of these.

1. Assessment: AI reverse-engineering of the legacy applications 

Our Mainframe Assessment Tool (MAT) reverse-engineers legacy codebases at enterprise scale to provide both explainability for the current legacy applications and sets the required foundation for modernization.

MAT delivers deep insights into your mainframe environment in four key areas:

1. Dependency visualization: Mapping relationships and interconnectivity between the different applications and data stores, such as DB2 databases or VSAM files.

2. Automated business rule extraction (BRE): Translating complex mainframe application logic into both plain-language requirements and visual decision trees.

3. Automated documentation: Generating comprehensive, up-to-date technical documentation directly from your production mainframe source code.

4. Domain and business function discovery: Automatically identifying application boundaries, grouping applications into high-level business domains and visualizing the architecture for these domains including inputs, outputs, interfaces, and where processing occurs.

MAT delivers the clean, verified logic requirements extracted from existing applications and business processes . By integrating these outputs directly into agentic modernization workflows through MCP, it equips your AI agents with the granular, application-specific context they need for high-accuracy code transformation, scaled execution, and optimized for your own codebase.

2. Modernization: code transformation 

Modernization requires strategic choices tailored to your desired business outcomes. There is rarely a single path that fits every use case. Our adaptable approach lets you select the exact depth of modernization your business needs, allowing you to apply different strategies to different mainframe workloads.

To bridge the gap between theory and execution (aka AI to Applied AI), we partnered with our Mainframe Modernization Professional Services team to build specialized AI agents. These agents directly codify their proven methodologies and hands-on field experience into structured, agentic modernization workflows.

With Google’s agentic mainframe modernization solution, you have two ways to modernize:

Rewrite / Reimagine

Choose this path for legacy applications where business logic innovation drives the highest strategic ROI. This pattern relies on the Mainframe Assessment Tool (MAT) to extract business rules together with our Mainframe Modernization Agents to handle forward-engineering. This agentic workflow analyzes and extracts complex business processes to translate legacy code into clear business specifications.

Combined with Antigravity as the agentic harness, this solution provides a safe, AI-accelerated development pipeline with optional human-in-the-loop governance at every step: business rule extraction from the mainframe applications -> creating target application specifications -> creating target architecture design -> generate user stories and backlog -> create the agentic coding implementation plan and more. 

This allows engineering teams to decouple complex logic from implementation details and replaces the legacy mainframe "black box" with a transparent, easily maintainable, and highly evolvable cloud-native applications.

Deterministic modernization (like-to-like)

This pattern is designed for use cases that require structural modernization while preserving exact, legacy application behavior. We use AI for direct code-to-code modernization of the internal application structures, using strict contract fidelity as our governing constraint. The modernized system must produce the identical business output as the legacy system for any given input, removing technical debt without altering external application interfaces.

Matching the pattern to the workload

What does modernizing with these two patterns look like in the real world? Imagine a customer in the financial services industry that applies a mixed approach across their estate. They could modernize stable, high-volume back-office batch jobs (like nightly statement processing) with the like-for-like path to reduce MIPS consumption with reduced risk and faster timelines. They might choose deterministic AI modernization for their core general ledger, modernizing the data structure to Google Cloud SQL for better analytics while preserving the regulatory compliance logic. Finally, for applications that are competitive differentiators, such as a customer-facing loan origination platform, they could deploy a rewrite-with-AI strategy, using Gemini to rewrite the application for real-time approvals, creating a true differentiator for the business.

3. The safety net: De-risk before going live 

To reduce go-live risk, Google Cloud  Dual Run processes real-world production workloads simultaneously across both your mainframe and your new Google Cloud environment. It automatically captures live mainframe transactions, runs them against your modern applications, and compares the outputs side-by-side (protocols, messages and changes to data). This continuous validation runs until you achieve complete logic and data equivalence, ensuring safety and zero operational disruption before you retire the legacy applications on the mainframe.

Global enterprises are already using Dual Run to eliminate migration risk and even secure the strict regulatory approvals needed for modernization in certain industries. Think of Dual Run as your production-grade insurance policy for mainframe modernization success.

4. Data migration: Modernize siloed mainframe data 

Finally, with the Google Cloud Mainframe Connector, you can copy data off the mainframe and into various Google Cloud services such as BigQuery, Spanner, Cloud SQL, Cloud Storage, and others. Mainframe Connector handles the codebase and data-type conversions, and can easily be integrated into existing ETL processes to iteratively copy data off the mainframe and onto Google Cloud. Mainframe Connector lets you both offload processing from the mainframe to support both the augmentation modernization pattern as well as analytics/data-warehousing. Now you can unlock siloed mainframe data to create new business functions in the cloud, all while reducing MIPS usage.

Combined, these four solutions demonstrate how targeted applied AI can help solve real-world modernization challenges for mainframe customers: understanding the existing business processes, modernizing the applications, de-risking before going live and modernizing the siloed data. 

Put us to the test

Mainframe modernization shouldn't require a leap of faith. Put our AI-accelerated approach to the test through a targeted pilot program designed to move your enterprise from uncertainty to execution.

Here is how we get started:

1. Automated codebase assessment: Run a Mainframe Assessment Tool scan on a target application to map hidden dependencies, visualize domain architecture, and extract plain-language business rules directly from the legacy code.

2. Agentic modernization workshop and pilot: Collaborate hands-on with Google Cloud experts and specialized partners to showcase the capabilities of how agentic workflows can be applied in the real-world to solve the mainframe modernization problem. Together we can pick one application to modernize and build the business case for modernization.

Ready to get started? Contact us at mainframe@google.com.