Table of contents
Introduction
Setting up a proxy inside of GCP can be more that simply including a VM runnig Squid or NGINX. If you want to control access properly, further restrictions should be applied and suitable configuration to agents, applications and VMs is required.
For example, blocking external access requires creating policies for public IP address usage on projects, removing/controlling routes and disabling private API access on subnets (not all APIs are available using Private Google Access).
Once this is done, key agents also won’t be able to perform their duties. The Stackdriver monitoring agent, Google’s version of fluentd and the Google SDK all need to talk to the API and will need to be configured to talk to the proxy.
GCP also gives a whole level of extra control of what your load balancer can do beyond a simple Amazon ELB. This is great for functionality but, as of writing, there is no simple “LB appliance” (one is planned), so it means there is much more to set up and understand. If you’re using a dual-homed VPN connection, there are some extra considerations that need to be made
Infrastructure
For a basic TCP proxy which is resilient for on-premise connections over redundant VPN legs, you will need the following components:
- Two VPN concentrators
- Two subnets
- Routes to different VPN concentrators
- Firewall rules for the proxy
- An internally facing forwarding rule
- A compute engine healthcheck
- A backend service
- Two managed instance group backends, for resilience. We could use multi-zone if we didn’t have to use different instance templates for the VPN.
- Two instance templates. Again, we could use 1 template if we didn’t have to assign different route tags for resilience over VPN
- An instance image, with a configured web proxy (we’ll use Squid in this example)
- A configured base image that engineers can use which has preconfigured agents and SDK
Let’s break this down to some minimum requirements
- VPN concentrators pointing to different DCs or offices
- E.g. vpn1 and vpn2
- Subnets subnet-1 and subnet-2
- With 10.0.1.0/24 for subnet-1 and 10.0.2.0/24 for subnet-2
- We will need route tags to both vpn1 and vpn2 and a route to the internet for our proxies (as of writing, this can only be done by instance tags).
- Our firewall rule tag should allow other instances to talk to our proxy on tcp port 3128
- Your forwarding rule needs to be given a name, region, internal IP address (as of writing, internal addresses cannot be assigned through the web interface), port and protocol.
- For example, my-forwarding-rule, europe-west1, 10.0.0.1, 3128, TCP
- Your healthcheck should have a name, protocol, port, interval, timeout, unhealthy threshold and healthy threshold.
- For example, my-healthcheck, TCP, 3128, 5 seconds, 3 seconds (must be less than interval), 2 consecutive failures mark unhealthy, 2 consecutive success mark healthy
- Your backend service needs to have a name, protocol, a timeout waiting for a backend to respond, healthcheck an affinity setting (sticky session) and two backend instance groups.
- For example, my-backend-service, TCP, 30 seconds, my-healthcheck, NONE, instance-group-zone-a, instance-group-zone-b.
- Your instance groups should have a name, zone, separate instance templates, auto-scaling enabled, a suitable scaling metric, minimum number of running instances, max number of instances (usually based on subnet limits) and a cooldown period dependent on how long your VM and application take to start up.
- For example, my-instance-group-zone-a, europe-west1-b my-instance-template-zone-a, autoscaling = on, cpu = 75%, 1 instance minimum, 12 instance maximum, 180 seconds
- Instance templates will need a name, suitable machine type, external IP (if any APIs you’re trying to connect to aren’t part of the private API access), use our Squid proxy image, one of our subnets, instance tags for our VPN route, internet, proxy firewall rule
- An image configured for Squid Proxy, IPTables, Stackdriver Agent, Google Fluentd and Google SDK
- An image configure basic services of Stackdriver Agent, Google Fluentd and Google SDK
Squid Proxy
Basic configuration that should be applied
# deny all other access to this proxy http_access deny all # Squid normally listens to port 3128 http_port 3128 # don't cache cache deny all
Some rules for internal subnets and specific hosts (these need to be before the http_access deny all
):
acl vpn1 src 10.0.1.0/24 # Datacentre/office 1 acl vpn2 src 10.0.2.0/24 # Datacentre/office 2 acl to_metadata dst 169.254.169.254 # GCP metadata acl some_internal_host src 192.168.0.2/32 # VM running on-premise acl googleapis dstdomain .googleapis.com .1e100.net .accounts.google.com .cloud.google.com # google domains to allow acl internet dst 0.0.0.0/0 http_access allow vpn1 googleapis http_access allow vpn2 googleapis http_access allow localhost http_access allow some_internal_host googleapis http_access deny to_metadata
IPTables
Below are the FW rules that match our Squid ACLs.
*filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [85:8271] -A INPUT -s 192.168.0.2/32 -p tcp -m tcp --dport 3128 -j ACCEPT -A INPUT -s 10.0.1.0/24 -p tcp -m tcp --dport 3128 -j ACCEPT -A INPUT -s 10.0.2.0/24 -p tcp -m tcp --dport 3128 -j ACCEPT -A INPUT -p tcp -m tcp --dport 3128 -j DROP -A OUTPUT -d 192.168.0.2/32 -p tcp -m tcp --sport 3128 -j ACCEPT -A OUTPUT -d 10.0.1.0/24 -p tcp -m tcp --sport 3128 -j ACCEPT -A OUTPUT -d 10.0.2.0/24 -p tcp -m tcp --sport 3128 -j ACCEPT -A OUTPUT -p tcp -m tcp --sport 3128 -j DROP COMMIT
Stackdriver Agent
Stackdriver needs the following line added to its configuration:
# url to a proxy for outbound https PROXY_URL="http://10.0.0.1:3128"
Google Fluentd
fluentd is a little more convoluted because there is no native proxying configuration. It also needs to discover what machine it is running on. Therefore, we have to use an HTTP proxy environment variable and a no proxy variable for the metadata server (so that it can work out what machine it is running on)
export HTTP_PROXY=http://10.0.0.1:3128 export HTTPS_PROXY=http://10.0.0.1:3128 export no_proxy=169.254.169.254,metadata,metadata.google.internal
Google SDK
The Google SDK is also a little different in that there are some processes which set up the VM initially and they look for the existence of a template file to make modifications to the Boto configuration.
[Boto] proxy = 10.0.0.1 proxy_port = 3128
When the instance setup scripts are ran sudo python /usr/lib/python2.7/site-packages/google_compute_engine/instance_setup/instance_setup.py
, this will be added to /etc/boto.cfg
# This file is automatically created at boot time by the /usr/lib/python # 2.7/site-packages/google_compute_engine/boto/boto_config.pyc script. # Do not edit this file directly. If you need to add items to this file, # create or edit /etc/boto.cfg.template instead and then re-run the # script. [Boto] proxy = 10.0.0.1 proxy_port = 3128 [GSUtil] default_project_id = 1234567890 default_api_version = 2 [GoogleCompute] service_account = default [Plugin] plugin_directory = /usr/lib/python2.7/site-packages/google_compute_engine/boto
GCP Load balancer
$gcloud compute health-checks create tcp my-healthcheck-googleapisproxy --description "Healthcheck for ensuring loadbalanced instances for googleapisproxy can have requests sent to them" --timeout 2 --check-interval 5 --healthy-threshold 2 --unhealthy-threshold 2 --port 3128 --port-name ndl-aas --proxy-header PROXY_V1 --request "" --response "" Created [https://www.googleapis.com/compute/v1/projects/my-project/global/healthChecks/my-healthcheck-googleapisproxy]. NAME PROTOCOL my-healthcheck-googleapisproxy TCP
$gcloud compute instance-templates create my-instance-template-googleapisproxy-vpn1 --image-family my-googleapisproxy-image-family --image-project my-project --description "Template for creating googleapisproxy instances WITH external access" --machine-type n1-highcpu-2 --metadata creator="me@example.com",environment=dev,technology=squid,application=googleapisproxy,version=3.5.20 --network projects/my-project/global/networks/my-network --subnet projects/my-project/regions/europe-west1/subnetworks/subnet-1 --tags vpn1-route-tag,internet-route-tag,proxy-fw-rule-tag Created [https://www.googleapis.com/compute/v1/projects/my-project/global/instanceTemplates/my-instance-template-googleapisproxy-vpn1]. NAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP my-instance-template-googleapisproxy-vpn1 n1-highcpu-2 2017-08-21T12:33:44
$gcloud compute instance-templates create my-instance-template-googleapisproxy-vpn2 --image-family my-googleapisproxy-image-family --image-project my-project --description "Template for creating googleapisproxy instances WITH external access" --machine-type n1-highcpu-2 --metadata creator="me@example.com",environment=dev,technology=squid,application=googleapisproxy,version=3.5.20 --network projects/my-project/global/networks/my-network --subnet projects/my-project/regions/europe-west1/subnetworks/subnet-2 --tags vpn2-route-tag,internet-route-tag,proxy-fw-rule-tag Created [https://www.googleapis.com/compute/v1/projects/my-project/global/instanceTemplates/my-instance-template-googleapisproxy-vpn2]. NAME MACHINE_TYPE PREEMPTIBLE CREATION_TIMESTAMP my-instance-template-googleapisproxy-vpn2 n1-highcpu-2 2017-08-21T12:33:44
$gcloud compute instance-groups managed create my-instance-group-managed-europe-west1-b-d-googleapisproxy --size 1 --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --zone europe-west1-b --template my-instance-template-googleapisproxy-vpn1 --base-instance-name my-googleapisproxy-vm Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/instanceGroupManagers/my-instance-group-managed-europe-west1-b-d-googleapisproxy]. NAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED my-instance-group-managed-europe-west1-b-d-googleapisproxy europe-west1-b zone my-europe-west1-b-d-googleapisproxy 0 1 my-instance-template-googleapisproxy-vpn1 no
$gcloud compute instance-groups managed create my-instance-group-managed-europe-west1-c-d-googleapisproxy --size 1 --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --zone europe-west1-c --template my-instance-template-googleapisproxy-vpn2 --base-instance-name my-googleapisproxy-vm Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/instanceGroupManagers/my-instance-group-managed-europe-west1-c-d-googleapisproxy]. NAME LOCATION SCOPE BASE_INSTANCE_NAME SIZE TARGET_SIZE INSTANCE_TEMPLATE AUTOSCALED my-instance-group-managed-europe-west1-c-d-googleapisproxy europe-west1-c zone my-europe-west1-c-d-googleapisproxy 0 1 my-instance-template-googleapisproxy-vpn2 no
$gcloud compute backend-services create my-backend-service-googleapisproxy --description "Backend service for load balancing across googleapisproxy instance groups" --health-checks my-healthcheck-googleapisproxy --session-affinity NONE --timeout 5m --protocol TCP --port-name ndl-aas --region europe-west1 --load-balancing-scheme INTERNAL Created [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/backendServices/my-backend-service-googleapisproxy]. NAME BACKENDS PROTOCOL my-backend-service-googleapisproxy TCP
$gcloud compute backend-services add-backend my-backend-service-googleapisproxy --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --instance-group my-instance-group-managed-europe-west1-b-d-googleapisproxy --instance-group-zone europe-west1-b --region europe-west1 Updated [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/backendServices/my-backend-service-googleapisproxy]. $gcloud compute backend-services add-backend my-backend-service-googleapisproxy --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --instance-group my-instance-group-managed-europe-west1-c-d-googleapisproxy --instance-group-zone europe-west1-c --region europe-west1 Updated [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/backendServices/my-backend-service-googleapisproxy].
$gcloud compute forwarding-rules create my-forwarding-rule-googleapisproxy --description "forwarding rule to direct network traffic to the googleapisproxy load balancer" --backend-service my-backend-service-googleapisproxy --address 10.0.0.1 --load-balancing-scheme INTERNAL --ip-protocol TCP --network projects/my-project/global/networks/my-network --subnet projects/my-project/regions/europe-west1/subnetworks/subnet-1 --ports 3128 --region europe-west1 Created [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/forwardingRules/my-forwarding-rule-googleapisproxy].
$gcloud compute instance-groups managed set-autoscaling my-instance-group-managed-europe-west1-c-d-googleapisproxy --description "Autoscaling policy for scaling up and down the number of instances in an instance group for googleapisproxy" --min-num-replicas 1 --max-num-replicas 4 --zone europe-west1-c --scale-based-on-cpu --target-cpu-utilization 0.75 --cool-down-period 180 Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/autoscalers/my-instance-group-managed-europe-west1-c-d-googleapisprox-rsjb]. --- autoscalingPolicy: coolDownPeriodSec: 180 cpuUtilization: utilizationTarget: 0.75 maxNumReplicas: 4 minNumReplicas: 1 creationTimestamp: '2017-08-21T12:33:44' description: Autoscaling policy for scaling up and down the number of instances in an instance group for googleapisproxy id: '123456789000000' kind: compute#autoscaler name: my-instance-group-managed-europe-west1-c-d-googleapisprox-rsjb selfLink: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/autoscalers/my-instance-group-managed-europe-west1-c-d-googleapisprox-rsjb status: ACTIVE target: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/instanceGroupManagers/my-instance-group-managed-europe-west1-c-d-googleapisproxy zone: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c $gcloud compute instance-groups managed set-autoscaling my-instance-group-managed-europe-west1-b-d-googleapisproxy --description "Autoscaling policy for scaling up and down the number of instances in an instance group for googleapisproxy" --min-num-replicas 1 --max-num-replicas 3 --zone europe-west1-b --scale-based-on-cpu --target-cpu-utilization 0.75 --cool-down-period 180 Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/autoscalers/my-instance-group-managed-europe-west1-b-d-googleapisprox-9610]. --- autoscalingPolicy: coolDownPeriodSec: 180 cpuUtilization: utilizationTarget: 0.75 maxNumReplicas: 3 minNumReplicas: 1 creationTimestamp: '2017-08-21T12:33:44' description: Autoscaling policy for scaling up and down the number of instances in an instance group for googleapisproxy id: '123456789011111' kind: compute#autoscaler name: my-instance-group-managed-europe-west1-b-d-googleapisprox-9610 selfLink: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/autoscalers/my-instance-group-managed-europe-west1-b-d-googleapisprox-9610 status: ACTIVE target: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/instanceGroupManagers/my-instance-group-managed-europe-west1-b-d-googleapisproxy zone: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b