An external HTTP(S) load balancer is a Google-recommended solution for distributing web traffic across multiple regions and zones, and providing high availability, scalability, and security for web applications. It supports both IPv4 and IPv6 addresses, and can handle SSL/TLS termination and encryption. It also integrates with Cloud CDN, Cloud Armor, and Cloud Identity-Aware Proxy for enhanced performance and protection. A managed instance group (MIG) can be used as a backend service for the HTTP(S) load balancer, and can automatically scale the number of VM instances based on the CPU load. A Cloud Storage bucket can also be used as a backend service for the HTTP(S) load balancer, and can serve static content such as images, videos, or HTML files. A URL map can be used to route requests to different backend services based on the path or host of the request. For example, a URL map can send requests for /static/* to the Cloud Storage bucket, and requests for /dynamic/* to the MIG. A managed SSL certificate can be used to secure the connection between the clients and the load balancer, and can be automatically provisioned and renewed by Google.

A is incorrect because an internal HTTP(S) load balancer is only visible within a VPC network, and not to the public internet. It is used for internal applications that need to communicate with other internal services. Identity-Aware Proxy is a service that provides secure access to web applications without using a VPN. It is not a load balancer, and it does not distribute user traffic.

B is incorrect because installing HTTPS certificates on the instance is not necessary, as the HTTP(S) load balancer can handle SSL/TLS termination and encryption. It is also more complex and less secure to manage the certificates on the instance level, as they need to be updated and synchronized across multiple instances.

D is incorrect because an external network load balancer is a TCP/UDP load balancer that operates at the network layer. It is not suitable for web applications that use HTTP(S) protocols, as it does not support SSL/TLS termination and encryption, URL maps, or Cloud Storage backends. It is also less efficient and scalable to forward the requests to the Cloud Storage from the web servers, as it adds an extra hop and latency.

HTTP(S) Load Balancing documentation

Setting up HTTP(S) Load Balancing with Cloud Storage

Creating and using SSL certificates

Choosing a load balancer

The goal is to enable teams to deploy infrastructure independently while ensuring compliance with company practices, without requiring teams to understand the underlying details of those practices.

Option A provides deployment capability but doesn't enforce practices. The Editor role is overly broad, and using the gcloud CLI directly requires knowledge of how to configure resources compliantly.

Option B requires teams to learn all the practices, contradicting the requirement that they don't need to know the implementation details.

Option C (Organization Policies) is useful for setting constraints (e.g., disallowing public IPs, restricting regions), but it doesn't provide pre-configured, deployable components that embody best practices. Teams still need to figure out how to build compliant resources within the policy constraints.

Option D (Terraform Modules): This approach encapsulates the company's required practices within reusable infrastructure-as-code modules. Engineering teams can then use these modules as building blocks, providing only the necessary input parameters (like application name orsize). The module handles the compliant implementation details internally. This allows teams to deploy independently and ensures compliance without needing deep knowledge of every practice.

Using standardized, compliant modules is a common pattern for enabling self-service infrastructure deployment while maintaining standards and governance.

https://www.cloudskillsboost.google/focuses/1035?parent=catalog#:~:text=custom%20role%20at%20the%20organization%20level

C:\GCP\appeng>gcloud config list

[core]

account = xxx@gmail.com

disable_usage_reporting = False

project = my-first-demo-xxxx

https://cloud.google.com/endpoints/docs/openapi/troubleshoot-gce-deployment

Cloud Functions is Google Cloud’s event-driven serverless compute platform that lets you run your code locally or in the cloud without having to provision servers. Cloud Functions scales upor down, so you pay only for compute resources you use. Cloud Functions have excellent integration with Cloud Pub/Sub, lets you scale down to zero and is recommended by Google as the ideal serverless platform to use when dependent on Cloud Pub/Sub.

"If you’re building a simple API (a small set of functions to be accessed via HTTP or Cloud Pub/Sub), we recommend using Cloud Functions."

Ref: https://cloud.google.com/serverless-options

Let's evaluate each option based on the requirement of sub-millisecond latency for globally stored IoT data:

A. Spanner with Caching: While Spanner offers strong consistency and global scalability, the base latency might not consistently be sub-millisecond for all read/write operations globally. Introducing caching adds complexity and doesn't guarantee sub-millisecond latency for all initial reads or cache misses.

B. Bigtable: Bigtable is a highly scalable NoSQL database service designed for low-latency, high-throughput workloads. It excels at storing and retrieving large volumes of time-series data, which is typical for IoT sensor data. Its architecture is optimized for single-key lookups and scans, providing consistent sub-millisecond latency, making it a strong candidate for this use case.

C. BigQuery: BigQuery is a fully managed, serverless data warehouse designed for analytical queries on large datasets. While it's excellent for analyzing IoT data in batch, it's not optimized for the low-latency, high-throughput ingestion and retrieval required for real-time IoT applications with sub-millisecond latency needs.

D. Cloud Storage with Cloud CDN: Cloud Storage is object storage and is not designed for low-latency transactional workloads. Cloud CDN is a content delivery network that caches contentcloser to users for faster delivery, but it's not suitable for the primary storage of rapidly incoming IoT sensor data requiring sub-millisecond write latency.

Google Cloud Documentation References:

Cloud Bigtable Overview: https://cloud.google.com/bigtable/docs/overview - This document highlights Bigtable's suitability for low-latency and high-throughput applications, including IoT. It mentions its ability to handle massive amounts of data with consistent performance.

Spanner Overview: https://cloud.google.com/spanner/docs/overview - While Spanner offers low latency, Bigtable is generally preferred for extremely high-throughput, low-latency use cases like raw sensor data ingestion due to its optimized architecture for such workloads.

BigQuery Overview: https://cloud.google.com/bigquery/docs/introduction - This emphasizes BigQuery's analytical capabilities rather than low-latency operational workloads.

Cloud Storage Overview: https://cloud.google.com/storage/docs/overview - This describes Cloud Storage as object storage, not ideal for sub-millisecond latency reads and writes required for real-time IoT data.

===========

Install the workload in a compute engine VM, start and stop the instance as needed, because as per the question the VM runs for 30 hours, process can be performed offline and should not be interrupted, if interrupted we need to restart the batch process again. Preemptible VMs are cheaper, but they will not be available beyond 24hrs, and if the process gets interrupted the preemptible VM will restart.

You write a lifecycle management rule in XML and push it to the bucket with gsutil lifecycle set config-xml-file. is not right.

gsutil lifecycle set enables you to set the lifecycle configuration on one or more buckets based on the configuration file provided. However, XML is not a valid supported type for the configuration file.

Ref: https://cloud.google.com/storage/docs/gsutil/commands/lifecycle

Write a script that runs gsutil ls -lr gs://myapp-gcp-ace-logs/ to find and remove items older than 90 days. Repeat this process every morning. is not right.

This manual approach is error-prone, time-consuming and expensive. GCP Cloud Storage provides lifecycle management rules that let you achieve this with minimal effort.

Write a script that runs gsutil ls -l gs://myapp-gcp-ace-logs/ to find and remove items older than 90 days. Schedule the script with cron. is not right.

This manual approach is error-prone, time-consuming and expensive. GCP Cloud Storage provides lifecycle management rules that let you achieve this with minimal effort.

Write a lifecycle management rule in JSON and push it to the bucket with gsutil lifecycle set config-json-file. is the right answer.

You can assign a lifecycle management configuration to a bucket. The configuration contains a set of rules which apply to current and future objects in the bucket. When an object meets the criteria of one of the rules, Cloud Storage automatically performs a specified action on the object. One of the supported actions is to Delete objects. You can set up a lifecycle management to delete objects older than 90 days. gsutil lifecycle set enables you to set the lifecycle configuration on the bucket based on the configuration file. JSON is the only supported type for the configuration file. The config-json-file specified on the command line should be a path to a local file containing the lifecycle configuration JSON document.

Ref: https://cloud.google.com/storage/docs/gsutil/commands/lifecycle

Ref: https://cloud.google.com/storage/docs/lifecycle

https://cloud.google.com/bigquery/docs/quickstarts/quickstart-web-ui?authuser=4

You can create two configurations – one for the development project and another for the production project. And you do that by running “gcloud config configurations create” command.

https://cloud.google.com/sdk/gcloud/reference/config/configurations/create

In your custom script, you can load these configurations one at a time and execute gcloud compute instances list to list Google Compute Engine instances in the project that is active in the gcloud configuration.

Ref: https://cloud.google.com/sdk/gcloud/reference/compute/instances/list

Once you have this information, you can export it in a suitable format to a suitable target e.g. export as CSV or export to Cloud Storage/BigQuery/SQL, etc

You can use Google Cloud Deployment Manager to create a set of Google Cloud resources and manage them as a unit, called a deployment. For example, if your team's development environment needs two virtual machines (VMs) and a BigQuery database, you can define these resources in a configuration file, and use Deployment Manager to create, change, or delete these resources. You can make the configuration file part of your team's code repository, so that anyone can create the same environment with consistent results.https://cloud.google.com/deployment-manager/docs/quickstart

This answer is the most effective way to validate whether the service account used by the CI/CD server has the appropriate roles in the specific project. By checking the IAM roles assigned to the service account, you can see which permissions the service account has and which resources it can access. You can also check if the service account inherits any roles from the folder or organization levels, which may affect its access to the project. You can use the Google Cloud console, the gcloud command-line tool, or the IAM API to view the IAM roles of a service account.

https://cloud.google.com/storage/docs/parallel-composite-uploads

https://cloud.google.com/storage/docs/uploads-downloads#parallel-composite-uploads

Keywords: - continually -> Standard - mission-critical analytics -> dual-regional

https://cloud.google.com/appengine/docs/flexible/managing-projects-apps-billing#:~:text=Each%20Cloud%20project%20can%20contain%20only%20a%20single%20App%20Engine%20application%2C%20and%20once%20created%20you%20cannot%20change%20the%20location%20of%20your%20App%20Engine%20application.

Two App engine can't be running on the same project: you can check this easy diagram for more info:https://cloud.google.com/appengine/docs/standard/an-overview-of-app-engine#components_of_an_application

And you can't change location after setting it for your app Engine.https://cloud.google.com/appengine/docs/standard/locations

App Engine is regional and you cannot change an apps region after you set it. Therefore, the only way to have an app run in another region is by creating a new project and targeting the app engine to run in the required region (asia-northeast1 in our case).

Ref: https://cloud.google.com/appengine/docs/locations

Ref: https://cloud.google.com/compute/docs/machine-types#n1_machine_type

https://cloud.google.com/logging/docs/audit

Instance groups are collections of virtual machine (VM) instances that you can manage as a single entity. Instance groups can help you simplify the management of multiple instances, reduce operational costs, and improve the availability and performance of your applications. Instance groups support autoscaling, which automatically adds or removes instances from thegroup based on increases or decreases in load. Autoscaling helps your applications gracefully handle increases in traffic and reduces cost when the need for resources is lower. You can set the autoscaling policy based on CPU utilization, load balancing capacity, Cloud Monitoring metrics, or a queue-based workload. In this case, since the video encoding software is CPU-intensive, setting the autoscaling based on CPU utilization is the best option to ensure high availability and optimal performance. References:

Instance groups

Autoscaling groups of instances

Forwarding zones Cloud DNS forwarding zones let you configure target name servers for specific private zones. Using a forwarding zone is one way to implement outbound DNS forwarding from your VPC network. A Cloud DNS forwarding zone is a special type of Cloud DNS private zone. Instead of creating records within the zone, you specify a set of forwarding targets. Each forwarding target is an IP address of a DNS server, located in your VPC network, or in an on-premises network connected to your VPC network by Cloud VPN or Cloud Interconnect.

https://cloud.google.com/nat/docs/overview

DNS configuration Your on-premises network must have DNS zones and records configured so that Google domain names resolve to the set of IP addresses for either private.googleapis.com or restricted.googleapis.com. You can create Cloud DNS managed private zones and use a Cloud DNS inbound server policy, or you can configure on-premises name servers. For example, you can use BIND or Microsoft Active Directory DNS.https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-domain

Navigate to Stackdriver Logging and select resource.labels.project_id=*. is not right.

Log entries are held in Stackdriver Logging for a limited time known as the retention period  which is 30 days (default configuration). After that, the entries are deleted. To keep log entries longer, you need to export them outside of Stackdriver Logging by configuring log sinks.

Ref: https://cloud.google.com/blog/products/gcp/best-practices-for-working-with-google-cloud-audit-logging

Configure a Cloud Scheduler job to read from Stackdriver and store the logs in BigQuery. Configure the table expiration to 60 days. is not right.

While this works, it makes no sense to use Cloud Scheduler job to read from Stackdriver and store the logs in BigQuery when Google provides a feature (export sinks) that does exactly the same thing and works out of the box.

Ref: https://cloud.google.com/logging/docs/export/configure_export_v2

Create a Stackdriver Logging Export with a Sink destination to Cloud Storage. Create a lifecycle rule to delete objects after 60 days. is not right.

You can export logs by creating one or more sinks that include a logs query and an export destination. Supported destinations for exported log entries are Cloud Storage, BigQuery, and Pub/Sub.

Ref: https://cloud.google.com/logging/docs/export/configure_export_v2

Sinks are limited to exporting log entries from the exact resource in which the sink was created: a Google Cloud project, organization, folder, or billing account. If it makes it easier to exporting from all projects of an organication, you can create an aggregated sink that can export log entries from all the projects, folders, and billing accounts of a Google Cloud organization.

Ref: https://cloud.google.com/logging/docs/export/aggregated_sinks

Either way, we now have the data in Cloud Storage, but querying logs information from Cloud Storage is harder than Querying information from BigQuery dataset. For this reason, we should prefer Big Query over Cloud Storage.

Create a Stackdriver Logging Export with a Sink destination to a BigQuery dataset. Configure the table expiration to 60 days. is the right answer.

You can export logs by creating one or more sinks that include a logs query and an export destination. Supported destinations for exported log entries are Cloud Storage, BigQuery, and Pub/Sub.

Ref: https://cloud.google.com/logging/docs/export/configure_export_v2

Sinks are limited to exporting log entries from the exact resource in which the sink was created: a Google Cloud project, organization, folder, or billing account. If it makes it easier to exporting from all projects of an organication, you can create an aggregated sink that can export log entries from all the projects, folders, and billing accounts of a Google Cloud organization.

Ref: https://cloud.google.com/logging/docs/export/aggregated_sinks

Either way, we now have the data in a BigQuery Dataset. Querying information from a Big Query dataset is easier and quicker than analyzing contents in Cloud Storage bucket. As our requirement is to Quickly analyze the log contents, we should prefer Big Query over Cloud Storage.

Also, You can control storage costs and optimize storage usage by setting the default table expiration for newly created tables in a dataset. If you set the property when the dataset is created, any table created in the dataset is deleted after the expiration period. If you set the property after the dataset is created, only new tables are deleted after the expiration period.

For example, if you set the default table expiration to 7 days, older data is automatically deleted after 1 week.

Ref: https://cloud.google.com/bigquery/docs/best-practices-storage

https://cloud.google.com/sql/docs/mysql/backup-recovery/backups#can_i_export_a_backup

https://cloud.google.com/sql/docs/mysql/import-export#automating_export_operations

To set a default cluster for gcloud commands, run the following command: gcloud config set container/cluster CLUSTER_NAMEhttps://cloud.google.com/kubernetes-engine/docs/how-to/managing-clusters?hl=en

Let's analyze each option based on the requirements: PostgreSQL compatibility, significant performance improvement, minimal schema/code changes, handling large data volumes, Google-recommended practices, and cost minimization:

A. Migrate your data to AlloyDB: AlloyDB for PostgreSQL is a fully managed, PostgreSQL-compatible database service that offers significant performance improvements over standard PostgreSQL due to its architectural optimizations. It is designed to handle large data volumes and minimizes the need for schema and application code changes as it's wire-compatible with PostgreSQL. This aligns well with the requirements for performance improvement, minimal changes, large data, and being a Google-recommended option for PostgreSQL workloads.

B. Migrate your data to Spanner: Spanner is a globally distributed, horizontally scalable database with strong consistency. While it offers excellent scalability and performance, it's not directly PostgreSQL-compatible. Migrating to Spanner would likely require significant schema and application code changes due to differences in data modeling and SQL dialect.

C. Migrate your data to Firebase: Firebase is a suite of mobile and web development tools, with its primary database offering being Firestore (a NoSQL document database) and Realtime Database. These are not PostgreSQL-compatible and would require substantial changes to the data model and application code.

D. Migrate your data to Bigtable: Bigtable is a highly scalable NoSQL wide-column store. It's not compatible with PostgreSQL and requires a completely different data model and application logic.

Therefore, AlloyDB is the most suitable option as it provides PostgreSQL compatibility for minimal migration effort, significant performance improvements, scalability for large data volumes, and is a recommended Google Cloud database service for PostgreSQL workloads.

Google Cloud Documentation References:

AlloyDB for PostgreSQL Overview: https://cloud.google.com/alloydb/docs/overview - This document highlights AlloyDB's PostgreSQL compatibility, performance benefits, scalability, and suitability for migrating existing PostgreSQL workloads.

Spanner Overview: https://cloud.google.com/spanner/docs/overview - This emphasizes Spanner's unique features and differences from traditional relational databases like PostgreSQL.

Firebase Documentation: https://firebase.google.com/docs - This outlines the features of Firebase, including Firestore and Realtime Database, highlighting their NoSQL nature and incompatibility with PostgreSQL.

Cloud Bigtable Overview: https://cloud.google.com/bigtable/docs/overview - This describes Bigtable as a NoSQL database, emphasizing its differences from relational databases like PostgreSQL.

===========

https://support.google.com/cloudidentity/answer/6262987?hl=en &ref_topic=7558767

The pending Pods resource requests are too large to fit on a single node of the cluster. Too many Pods are already running in the cluster, and there are not enough resources left to schedule the pending Pod. is the right answer.

When you have a deployment with some pods in running and other pods in the pending state, more often than not it is a problem with resources on the nodes. Heres a sample output of this use case. We see that the problem is with insufficient CPU on the Kubernetes nodes so we have to either enable auto-scaling or manually scale up the nodes.

Keyword, Version, traffic splitting, App Engine supports traffic splitting for versions before releasing.

https://cloud.google.com/bigquery/docs/access-control#custom_roles

Custom roles enable you to enforce the principle of least privilege, ensuring that the user and service accounts in your organization have only the permissions essential to performing their intended functions.

Logged onto console and followed the steps and was able to see all the assigned users and roles.

Before you run the gcloud compute instances list command, you need to do two things: authenticate with your user account and set the default project for gcloud CLI.

To authenticate with your user account, you need to run gcloud auth login, enter your login credentials in the dialog window, and paste the received login token to gcloud CLI. This will authorize the gcloud CLI to access Google Cloud resources on your behalf1.

To set the default project for gcloud CLI, you need to run gcloud config set project $my_project, where $my_project is the ID of the project that contains the instances you want to list. This will saveyou from having to specify the project flag for every gcloud command2.

Option B is not recommended, because using a service account key increases the risk of credential leakage and misuse. It is also not necessary, because you can use your user account to authenticate to the gcloud CLI3. Option C is not correct, because there is no such thing as a Cloud Identity user account key. Cloud Identity is a service that provides identity and access management for Google Cloud users andgroups4. Option D is not required, because the gcloud compute instances list command does not depend on the default zone. You can list instances from all zones or filter by a specific zone using the --filter flag.

1: https://cloud.google.com/sdk/docs/authorizing

2: https://cloud.google.com/sdk/gcloud/reference/config/set

3: https://cloud.google.com/iam/docs/best-practices-for-managing-service-account-keys

4: https://cloud.google.com/identity/docs/overview

https://cloud.google.com/sdk/gcloud/reference/compute/instances/list

https://cloud.google.com/iam/docs/understanding-roles#billing-roles

Types of audit logs Cloud Audit Logs provides the following audit logs for each Cloud project, folder, and organization: Admin Activity audit logs Data Access audit logs System Event audit logs Policy Denied audit logs *Data Access audit logs contain API calls that read the configuration or metadata of resources, as well as user-driven API calls that create, modify, or read user-provided resource data.https://cloud.google.com/logging/docs/audit#types

https://cloud.google.com/logging/docs/audit#data-access Cloud Storage: When Cloud Storage usage logs are enabled, Cloud Storage writes usage data to the Cloud Storagebucket, which generates Data Access audit logs for the bucket. The generated Data Access audit log has its caller identity redacted.

https://cloud.google.com/resource-manager/docs/project-migration#oauth_consent_screen

https://cloud.google.com/resource-manager/docs/project-migration

Cloud Storage allows world-wide storage and retrieval of any amount of data at any time. We dont need to set up auto-scaling ourselves. Cloud Storage autoscaling is managed by GCP. Cloud Storage is an object store so it is suitable for storing photos. Cloud Storage allows world-wide storage and retrieval so cater well to our worldwide audience. Cloud storage provides us lifecycle rules that can be configured to automatically delete objects older than 30 days. This also fits our requirements. Finally, Google Cloud Storage offers several storage classes such as Nearline Storage ($0.01 per GB per Month) Coldline Storage ($0.007 per GB per Month) and Archive Storage ($0.004 per GB per month) which are significantly cheaper than any of the options above.

Ref: https://cloud.google.com/storage/docs

Ref: https://cloud.google.com/storage/pricing

https://cloud.google.com/billing/docs/how-to/export-data-bigquery "Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify."

Signed URLs are used to give time-limited resource access to anyone in possession of the URL, regardless of whether they have a Google account.https://cloud.google.com/storage/docs/access-control/signed-urls

Understanding IAM custom roles

Key Point: Custom roles enable you to enforce the principle of least privilege, ensuring that the user and service accounts in your organization have only the permissions essential to performing their intended functions.

Basic concepts

Custom roles are user-defined, and allow you to bundle one or more supported permissions to meet your specific needs. Custom roles are not maintained by Google; when new permissions, features, or services are added to Google Cloud, your custom roles will not be updated automatically.

When you create a custom role, you must choose an organization or project to create it in. You can then grant the custom role on the organization or project, as well as any resources within that organization or project.

https://cloud.google.com/iam/docs/understanding-custom-roles#basic_concepts

Text Description automatically generated with low confidence

Cloud Functions is Google Cloud’s event-driven serverless compute platform. It automatically scales based on the load and requires no additional configuration. You pay only for the resources used.

Ref: https://cloud.google.com/functions

While all other options i.e. Google Compute Engine, Google Kubernetes Engine, Google App Engine support autoscaling, it needs to be configured explicitly based on the load and is not as trivial as the scale up or scale down offered by Google’s cloud functions.

https://cloud.google.com/architecture/best-practices-vpc-design#shared-service Cloud VPN is another alternative. Because Cloud VPN establishes reachability through managed IPsec tunnels, it doesn't have the aggregate limits of VPC Network Peering. Cloud VPN uses a VPN Gateway for connectivity and doesn't consider the aggregate resource use of the IPsec peer. The drawbacks of Cloud VPN include increased costs (VPN tunnels and traffic egress), management overhead required to maintain tunnels, and the performance overhead of IPsec.

(A) Object Lifecycle Management

Delete

The Delete action deletes an object when the object meets all conditions specified in the lifecycle rule.

Exception: In buckets with Object Versioning enabled, deleting the live version of an object causes it to become a noncurrent version, while deleting a noncurrent version deletes that version permanently.

https://cloud.google.com/storage/docs/lifecycle#delete

(B) Signed URLs

This page provides an overview of signed URLs, which you use to give time-limited resource access to anyone in possession of the URL, regardless of whether they have a Google account

https://cloud.google.com/storage/docs/access-control/signed-urls

The reason is that when you do health check, you want the VM to be working. Do the first check after initial setup time of 3 mins = 180 s < 200 s is reasonable.

The reason why our autoscaling is adding more instances than needed is that it checks 30 seconds after launching the instance and at this point, the instance isnt up and isnt ready to serve traffic. So our autoscaling policy starts another instance  again checks this after 30 seconds and the cycle repeats until it gets to the maximum instances or the instances launched earlier are healthy and start processing traffic  which happens after 180 seconds (3 minutes). This can be easily rectified by adjusting the initial delay to be higher than the time it takes for the instance to become available for processing traffic.So setting this to 200 ensures that it waits until the instance is up (around 180-second mark) and then starts forwarding traffic to this instance. Even after a cool out period, if the CPU utilization is still high, the autoscaler can again scale up but this scale-up is genuine and is based on the actual load.

Initial Delay Seconds  This setting delays autohealing from potentially prematurely recreating the instance if the instance is in the process of starting up. The initial delay timer starts when the currentAction of the instance is VERIFYING.

Ref: https://cloud.google.com/compute/docs/instance-groups/autohealing-instances-in-migs

"Billing Account Viewer access would usually be granted to finance teams, it provides access to spend information, but does not confer the right to link or unlink projects or otherwise manage the properties of the billing account."https://cloud.google.com/billing/docs/how-to/billing-access

As to mexblood1's point, CPU utilization is a recommended proxy for traffic when it comes to Cloud Spanner. See: Alerts for high CPU utilization The following table specifies our recommendations for maximum CPU usage for both single-region and multi-region instances. These numbers are to ensure that your instance has enough compute capacity to continue to serve your traffic in the event of the loss of an entire zone (for single-region instances) or an entire region (for multi-region instances). -https://cloud.google.com/spanner/docs/cpu-utilization

A tag is simply a character string added to a tags field in a resource, such as Compute Engine virtual machine (VM) instances or instance templates. A tag is not a separateresource, so you cannot create it separately. All resources with that string are considered to have that tag. Tags enable you to make firewall rules and routes applicable to specific VM instances.

https://cloud.google.com/sdk/gcloud

https://cloud.google.com/sdk/docs/configurations#multiple_configurations

The Datastore emulator provides local emulation of the production Datastore environment. You can use the emulator to develop and test your application locallyRef: https://cloud.google.com/datastore/docs/tools/datastore-emulator

A simple approach would be to use the command flags available when listing all the IAM policy for a given project. For instance, the following command: `gcloud projects get-iam-policy $PROJECT_ID --flatten="bindings[].members" --format="table(bindings.members)" --filter="bindings.role:roles/owner"` outputs all the users and service accounts associatedwith the role ‘roles/owner’ in the project in question.https://groups.google.com/g/google-cloud-dev/c/Z6sZs7TvygQ?pli=1

This is the most optimal solution. Rather than recreating all nodes, you create a new node pool with GPU enabled. You then modify the pod specification to target particular GPU types by adding node selector to your workloads Pod specification. YOu still have a single cluster so you pay Kubernetes cluster management fee for just one cluster thus minimizing the cost.

Ref: https://cloud.google.com/kubernetes-engine/docs/how-to/gpus

Ref: https://cloud.google.com/kubernetes-engine/pricing

Example:

apiVersion: v1

kind: Pod

metadata:

name:my-gpu-pod

spec:

containers:

name:my-gpu-container

image: nvidia/cuda:10.0-runtime-ubuntu18.04

command: [/bin/bash]

resources:

limits:

nvidia.com/gpu: 2

nodeSelector:

cloud.google.com/gke-accelerator: nvidia-tesla-k80# or nvidia-tesla-p100 or nvidia-tesla-p4 or nvidia-tesla-v100 or nvidia-tesla-t4

gcloud compute networks subnets expand-ip-range NAME gcloud compute networks subnets expand-ip-range - expand the IP range of a Compute Engine subnetworkhttps://cloud.google.com/sdk/gcloud/reference/compute/networks/subnets/expand-ip-range

Configure the ACLs on each image in Cloud Storage to give read-only access to the default Compute Engine service account. is not right.

As mentioned above, Container Registry ignores permissions set on individual objects within the storage bucket so this isnt going to work.

Ref: https://cloud.google.com/container-registry/docs/access-control

Comprehensive and Detailed In Depth Explanation:

The core requirements are secure access to a specific BigQuery dataset from a Compute Engine instance, using short-lived credentials, adhering to Google's best practices, and minimizing operational overhead.

A. Project-level IAM role: Granting the BigQuery Data Viewer role at the project level gives the service account broad access to all BigQuery datasets within that project. This violates the principle of least privilege, a fundamental security best practice, as the application should only have access to the designated dataset.

B. Hourly new service account with dataset-level role: While this aims to achieve short-lived credentials, the operational burden of creating, attaching, and managing IAM policies for a new service account every hour is significant and not a Google-recommended practice for routine access. It introduces unnecessary complexity and potential for errors.

C. Custom service account with dataset-level IAM role: This is the recommended and most efficient approach. You create a dedicated Google Cloud service account specifically for this application. You then grant this service account the necessary IAM role (e.g., BigQuery Data Viewer, or a more specific custom role) directly on the target BigQuery dataset. When the Compute Engine instance runs as this service account, the Google Cloud client libraries automatically handle the acquisition and rotation of short-lived OAuth 2.0 access tokens from the instance's metadata server. This eliminates the need to manage long-lived credentials (like service account keys) and ensures the application only has access to the intended dataset. This adheres to the principle of least privilege and minimizes operational costs.

D. Hourly new service account with project-level role: This option combines the high operational overhead of frequently creating new service accounts with the security risk of granting overly permissive project-level access. It is not a recommended practice.

Therefore, the most secure, cost-effective, and operationally efficient solution is to create a custom service account, attach it to the Compute Engine instance, and grant it the appropriateBigQuery IAM role specifically on the target dataset. The platform handles the short-lived credentials automatically.

Google Cloud Documentation References:

Creating and enabling service accounts for instances: https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances - Explains how to create and associate service accounts with Compute Engine VMs.  

Granting, changing, and revoking access to resources: https://cloud.google.com/iam/docs/granting-changing-revoking-access - Details how to manage IAM policies and grant roles to service accounts on specific resources (like BigQuery datasets).  

BigQuery IAM roles: https://cloud.google.com/bigquery/docs/control-access - Provides information on the various IAM roles available for controlling access to BigQuery resources at different levels (project, dataset, table, etc.).

Best practices for using service accounts: https://cloud.google.com/iam/docs/best-practices-for-using-service-accounts - Emphasizes the importance of the principle of least privilege and avoiding the management of long-lived service account keys when possible (relying on the metadata server for short-lived tokens).

Showing Deployment Manager templates to your team will allow you to define the changes you want to implement in your cloud infrastructure. You can use Cloud Source Repositories to store Deployment Manager templates and collaborate with your team. Cloud Source Repositories are fully-featured, scalable, and private Git repositories you can use to store, manage and track changes to your code.

https://cloud.google.com/source-repositories/docs/features

https://cloud.google.com/sdk/gcloud/reference/projects/create

Quickstart: Creating a New Instance Using the Command Line

Before you begin

1. In the Cloud Console, on the project selector page, select or create a Cloud project.

2. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

To use the gcloud command-line tool for this quickstart, you must first install and initialize the Cloud SDK:

1. Download and install the Cloud SDK using the instructions given on Installing Google Cloud SDK.

2. Initialize the SDK using the instructions given on Initializing Cloud SDK.

To use gcloud in Cloud Shell for this quickstart, first activate Cloud Shell using the instructions given on Starting Cloud Shell.

https://cloud.google.com/ai-platform/deep-learning-vm/docs/quickstart-cli#before-you-begin

gcloud compute ssh ensures that the user’s public SSH key is present in the project’s metadata. If the user does not have a public SSH key, one is generated using ssh-keygen and added to the project’s metadata. This is similar to the other option where we copy the key explicitly to the project’s metadata but here it is done automatically for us. There are also security benefits with this approach. When we use gcloud compute ssh to connect to Linux instances, we are adding a layer of security by storing your host keys as guest attributes. Storing SSH host keys as guest attributes improve the security of your connections by helping to protect against vulnerabilities such as man-in-the-middle (MITM) attacks. On the initial boot of a VM instance, if guest attributes are enabled, Compute Engine stores your generated host keys as guest attributes.

Compute Engine then uses these host keys that were stored during the initial boot to verify all subsequent connections to the VM instance.

Ref: https://cloud.google.com/compute/docs/instances/connecting-to-instance

Ref: https://cloud.google.com/sdk/gcloud/reference/compute/ssh

"The Cloud Spanner to Cloud Storage Text template is a batch pipeline that reads in data from a Cloud Spanner table, optionally transforms the data via a JavaScript User Defined Function (UDF) that you provide, and writes it to Cloud Storage as CSV text files."

https://cloud.google.com/dataflow/docs/guides/templates/provided-batch#cloudspannertogcstext

"The Dataflow connector for Cloud Spanner lets you read data from and write data to Cloud Spanner in a Dataflow pipeline"

https://cloud.google.com/spanner/docs/dataflow-connector

https://cloud.google.com/bigquery/external-data-sources

The goal is to create a serverless, scalable, and asynchronous transaction processing system with minimal management overhead.

Serverless Requirement:Options involving installing Kafka on VMs (A, B) or using VM instances for processing (B, C) introduce management overhead associated with VMs (patching, scaling configuration, OS management) and Kafka cluster management, violating the serverless and minimal management criteria.

Asynchronous Requirement:Both Kafka and Pub/Sub can handle asynchronous messaging. However, Pub/Sub is Google Cloud's fully managed, serverless messaging service, inherently minimizing management overhead compared to self-managed Kafka on VMs.

Scalability and Processing:Cloud Run is a fully managed, serverless platform that automatically scales based on traffic, suitable for processing transactions without managing underlying infrastructure. VM instances require manual scaling configuration or managed instance groups, adding overhead.

Combining Pub/Sub for asynchronous message ingestion (fully managed, serverless) and Cloud Run for processing (fully managed, serverless, scalable) directly meets all requirements: serverless, scalable, asynchronous, and minimal management overhead. Option D is the only one that uses fully serverless components for both ingestion and processing.

https://cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups#max_unavailable

https://cloud.google.com/compute/docs/instances/managing-instance-access

Cloud Spanner is a relational database and is highly scalable. Cloud Spanner is a highly scalable, enterprise-grade, globally-distributed, and strongly consistent database service built for the cloud specifically to combine the benefits of relational database structure with a non-relational horizontal scale. This combination delivers high-performance transactions and strong consistency across rows, regions, and continents with an industry-leading 99.999% availability SLA, no planned downtime, and enterprise-grade security

Ref: https://cloud.google.com/spanner

Graphical user interface, application, Teams Description automatically generated

Comprehensive and Detailed Explanation From Exact Extract:

The goal is to create a landing zone facilitating private IP communication across production projects and apply organization-wide firewall rules, following best practices and minimizing operational costs.

Network Structure:Individual VPCs with Peering (A, B): While VPC Peering allows private connectivity, managing a full mesh or complex peering topology across many projects becomes operationally complex and can hit peering limits. It's not the recommended pattern for centralized connectivity in a landing zone.

Shared VPC (C, D): This is the Google-recommended practice for scenarios where resources from multiple projects need to communicate privately within a common VPC network. A central host project owns the network, and service projects use it. This simplifies network administration and connectivity.

Firewall Rules:Organization Policies (A, C): These enforce organizational constraints (e.g., disable external IPs, restrict locations) but do not define specific network firewall rules (like allowing TCP ports).

Hierarchical Firewall Policies (B, D): These allow defining firewall rules at the Organization or Folder level, which are inherited by resources in descendant projects/folders. This is the mechanism to apply consistent firewall rules (like allowing specific TCP ports) across all VMs inthe organization (or a specific folder) efficiently, without managing rules in each individual VPC or project.

Combining Shared VPC for the network structure (best practice for cross-project private communication and central management) with Hierarchical Firewall Policies (for applying organization-wide firewall rules) meets all requirements efficiently and follows Google recommendations.  

they require cloud storage for archival and the want to automate the process to upload new medical image to cloud storage, hence we go for gsutil to copy on-prem images to cloud storage and automate the process via cron job. whereas Pub/Sub listens to the changes in the Cloud Storage bucket and triggers the pub/sub topic, which is not required.

gcloud deployment-manager deployments create creates deployments based on the configuration file. (Infrastructure as code). All the configuration related to the artifacts is in the configuration file. This command correctly creates a cluster based on the provided cluster.yaml configuration file.

Ref: https://cloud.google.com/sdk/gcloud/reference/deployment-manager/deployments/create

https://medium.com/google-cloud/how-to-deploy-cassandra-and-connect-on-google-cloud-platform-with-a-few-clicks-11ee3d7001d1

https://cloud.google.com/blog/products/databases/open-source-cassandra-now-managed-on-google-cloud

https://cloud.google.com/marketplace

You can deploy Cassandra as a Service, called Astra, on the Google Cloud Marketplace. Not only do you get a unified bill for all GCP services, your Developers can now create Cassandra clusters on Google Cloud in minutes and build applications with Cassandra as a database as a service without the operational overhead of managing Cassandra

Creating scheduled snapshots for persistent disk This document describes how to create a snapshot schedule to regularly and automatically back up your zonal and regional persistent disks. Use snapshot schedules as a best practice to back up your Compute Engine workloads. After creating a snapshot schedule, you can apply it to one or more persistent disks.https://cloud.google.com/compute/docs/disks/scheduled-snapshots

This aligns with Googles recommended practices. By creating a new project, we achieve complete isolation between development and production environments; as well as isolate this production application from production applications of other departments.

Ref: https://cloud.google.com/docs/enterprise/best-practices-for-enterprise-organizations#define-hierarchy

If your apps are fault-tolerant and can withstand possible instance preemptions, then preemptible instances can reduce your Compute Engine costs significantly. For example, batch processing jobs can run on preemptible instances. If some of those instances stop during processing, the job slows but does not completely stop. Preemptible instances complete your batch processing tasks without placing additional workload on your existing instances and without requiring you to pay full price for additional normal instances.

https://cloud.google.com/compute/docs/instances/preemptible

From this source:https://cloud.google.com/billing/docs/how-to/custom-roles#permission_association_and_inheritance

"For example, associating a project with a billing account requires the billing.resourceAssociations.create permission on the billing account and also the resourcemanager.projects.createBillingAssignment permission on the project. This is because project permissions are required for actions where project owners control access, while billing account permissions are required for actions where billing account administrators control access. When both should be involved, both permissions are necessary."

https://cloud.google.com/spanner/docs/cpu-utilization#recommended-max

https://cloud.google.com/appengine/docs/standard/python/splitting-traffic#gcloud

https://cloud.google.com/architecture/identity/migrating-consumer-accounts

IP 10.0.3.21 is internal by default, and to ensure that it will be static non-changing it should be selected as static internal ip address.

Installing Jenkins

In this section, you use Cloud Marketplace to provision a Jenkins instance. You customize this instance to use the agent image you created in the previous section.

Go to the Cloud Marketplace solution for Jenkins.

Click Launch on Compute Engine.

Change the Machine Type field to 4 vCPUs 15 GB Memory, n1-standard-4.

Machine type selection for Jenkins deployment.

Click Deploy and wait for your Jenkins instance to finish being provisioned. When it is finished, you will see:

Jenkins has been deployed.

https://cloud.google.com/solutions/using-jenkins-for-distributed-builds-on-compute-engine#installing_jenkins