Loading...
 

Setting up a network proxy in GCP

Introduction


Setting up a proxy inside of GCP can be more that simply including a VM runnig Squid or NGINX. If you want to control access properly, further restrictions should be applied and suitable configuration to agents, applications and VMs is required.

For example, blocking external access requires creating policies for public IP address usage on projects, removing/controlling routes and disabling private API access on subnets (not all APIs are available using Private Google Access).

Once this is done, key agents also won’t be able to perform their duties. The Stackdriver monitoring agent, Google’s version of fluentd and the Google SDK all need to talk to the API and will need to be configured to talk to the proxy.
 
GCP also gives a whole level of extra control of what your load balancer can do beyond a simple Amazon ELB. This is great for functionality but, as of writing, there is no simple “LB appliance” (one is planned),  so it means there is much more to set up and understand. If you’re using a dual-homed VPN connection, there are some extra considerations that need to be made

Infrastructure

For a basic TCP proxy which is resilient for on-premise connections over redundant VPN legs, you will need the following components:

  1. Two VPN concentrators
  2. Two subnets
  3. Routes to different VPN concentrators
  4. Firewall rules for the proxy
  5. An internally facing forwarding rule
  6. A compute engine healthcheck
  7. A backend service
  8. Two managed instance group backends, for resilience. We could use multi-zone if we didn’t have to use different instance templates for the VPN.
  9. Two instance templates. Again, we could use 1 template if we didn’t have to assign different route tags for resilience over VPN
  10. An instance image, with a configured web proxy (we’ll use Squid in this example)
  11. A configured base image that engineers can use which has preconfigured agents and SDK


Let’s break this down to some minimum requirements

  1. VPN concentrators pointing to different DCs or offices
    1. E.g. vpn1 and vpn2
  2. Subnets subnet-1 and subnet-2
    1. With 10.0.1.0/24 for subnet-1 and 10.0.2.0/24 for subnet-2
  3. We will need route tags to both vpn1 and vpn2 and a route to the internet for our proxies (as of writing, this can only be done by instance tags).
  4. Our firewall rule tag should allow other instances to talk to our proxy on tcp port 3128
  5. Your forwarding rule needs to be given a name, region, internal IP address (as of writing, internal addresses cannot be assigned through the web interface), port and protocol.
    1. For example, my-forwarding-rule, europe-west1, 10.0.0.1, 3128, TCP
  6. Your healthcheck should have a name, protocol, port, interval, timeout, unhealthy threshold and healthy threshold. 
    1. For example, my-healthcheck, TCP, 3128, 5 seconds, 3 seconds (must be less than interval), 2 consecutive failures mark unhealthy, 2 consecutive success mark healthy
  7. Your backend service needs to have a name, protocol, a timeout waiting for a backend to respond, healthcheck an affinity setting (sticky session) and two backend instance groups.
    1. For example, my-backend-service, TCP, 30 seconds, my-healthcheck, NONE, instance-group-zone-a, instance-group-zone-b.
  8. Your instance groups should have a name, zone, separate instance templates, auto-scaling enabled, a suitable scaling metric, minimum number of running instances, max number of instances (usually based on subnet limits) and a cooldown period dependent on how long your VM and application take to start up.
    1. For example, my-instance-group-zone-a, europe-west1-b my-instance-template-zone-a, autoscaling = on, cpu = 75%, 1 instance minimum, 12 instance maximum, 180 seconds
  9. Instance templates will need a name, suitable machine type, external IP (if any APIs you’re trying to connect to aren’t part of the private API access), use our Squid proxy image, one of our subnets, instance tags for our VPN route, internet, proxy firewall rule
  10. An image configured for Squid Proxy, IPTables, Stackdriver Agent, Google Fluentd and Google SDK
  11. An image configure basic services of Stackdriver Agent, Google Fluentd and Google SDK


 

Squid Proxy

Basic configuration that should be applied

/etc/squid/squid.conf
# deny all other access to this proxy
http_access deny all
# Squid normally listens to port 3128
http_port 3128
# don't cache
cache deny all


Some rules for internal subnets and specific hosts (these need to be before the http_access deny all):

/etc/squid/squid.conf
acl  vpn1   src  10.0.1.0/24 # Datacentre/office 1
acl  vpn2  src 10.0.2.0/24  # Datacentre/office 2
acl  to_metadata  dst 169.254.169.254 # GCP metadata
acl  some_internal_host  src  192.168.0.2/32 # VM running on-premise
acl  googleapis dstdomain .googleapis.com .1e100.net .accounts.google.com .cloud.google.com # google domains to allow
acl  internet dst  0.0.0.0/0

http_access allow vpn1 googleapis
http_access allow vpn2 googleapis
http_access allow localhost
http_access allow some_internal_host googleapis
http_access deny to_metadata


IPTables

Below are the FW rules that match our Squid ACLs.

/etc/sysconfig/stackdriver
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [85:8271]
-A INPUT -s 192.168.0.2/32 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -s 10.0.1.0/24 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -s 10.0.2.0/24 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3128 -j DROP
-A OUTPUT -d 192.168.0.2/32 -p tcp -m tcp --sport 3128 -j ACCEPT
-A OUTPUT -d 10.0.1.0/24 -p tcp -m tcp --sport 3128 -j ACCEPT
-A OUTPUT -d 10.0.2.0/24 -p tcp -m tcp --sport 3128 -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 3128 -j DROP
COMMIT

Stackdriver Agent

Stackdriver needs the following line added to its configuration:

/etc/sysconfig/iptables
# url to a proxy for outbound https
PROXY_URL="http://10.0.0.1:3128"

Google Fluentd

fluentd is a little more convoluted because there is no native proxying configuration. It also needs to discover what machine it is running on. Therefore, we have to use an HTTP proxy environment variable and a no proxy variable for the metadata server (so that it can work out what machine it is running on)

/etc/sysconfig/google-fluentd
export HTTP_PROXY=http://10.0.0.1:3128
export HTTPS_PROXY=http://10.0.0.1:3128
export no_proxy=169.254.169.254,metadata,metadata.google.internal

Google SDK

The Google SDK is also a little different in that there are some processes which set up the VM initially and they look for the existence of a template file to make modifications to the Boto configuration.

/etc/boto.cfg.template
[Boto]
proxy = 10.0.0.1
proxy_port = 3128


When the instance setup scripts are ran sudo python /usr/lib/python2.7/site-packages/google_compute_engine/instance_setup/instance_setup.py, this will be added to /etc/boto.cfg

/etc/boto.cfg
# This file is automatically created at boot time by the /usr/lib/python
# 2.7/site-packages/google_compute_engine/boto/boto_config.pyc script.
# Do not edit this file directly. If you need to add items to this file,
# create or edit /etc/boto.cfg.template instead and then re-run the
# script.

[Boto]
proxy = 10.0.0.1
proxy_port = 3128

[GSUtil]
default_project_id = 1234567890
default_api_version = 2

[GoogleCompute]
service_account = default

[Plugin]
plugin_directory = /usr/lib/python2.7/site-packages/google_compute_engine/boto

GCP Load balancer

Healthcheck
$gcloud compute health-checks create tcp  my-healthcheck-googleapisproxy --description "Healthcheck for ensuring loadbalanced instances for googleapisproxy can have requests sent to them" --timeout 2 --check-interval 5 --healthy-threshold 2 --unhealthy-threshold 2 --port 3128 --port-name ndl-aas --proxy-header PROXY_V1 --request "" --response ""
Created [https://www.googleapis.com/compute/v1/projects/my-project/global/healthChecks/my-healthcheck-googleapisproxy].
NAME                             PROTOCOL
my-healthcheck-googleapisproxy  TCP
Template for VPN1
$gcloud compute instance-templates create my-instance-template-googleapisproxy-vpn1 --image-family my-googleapisproxy-image-family --image-project my-project --description "Template for creating googleapisproxy instances WITH external access" --machine-type n1-highcpu-2 --metadata creator="me@example.com",environment=dev,technology=squid,application=googleapisproxy,version=3.5.20 --network projects/my-project/global/networks/my-network --subnet projects/my-project/regions/europe-west1/subnetworks/subnet-1 --tags vpn1-route-tag,internet-route-tag,proxy-fw-rule-tag
Created [https://www.googleapis.com/compute/v1/projects/my-project/global/instanceTemplates/my-instance-template-googleapisproxy-vpn1].
NAME                                       MACHINE_TYPE  PREEMPTIBLE  CREATION_TIMESTAMP
my-instance-template-googleapisproxy-vpn1  n1-highcpu-2               2017-08-21T12:33:44
Template for VPN2
$gcloud compute instance-templates create my-instance-template-googleapisproxy-vpn2 --image-family my-googleapisproxy-image-family --image-project my-project --description "Template for creating googleapisproxy instances WITH external access" --machine-type n1-highcpu-2 --metadata creator="me@example.com",environment=dev,technology=squid,application=googleapisproxy,version=3.5.20 --network projects/my-project/global/networks/my-network --subnet projects/my-project/regions/europe-west1/subnetworks/subnet-2 --tags vpn2-route-tag,internet-route-tag,proxy-fw-rule-tag
Created [https://www.googleapis.com/compute/v1/projects/my-project/global/instanceTemplates/my-instance-template-googleapisproxy-vpn2].
NAME                                       MACHINE_TYPE  PREEMPTIBLE  CREATION_TIMESTAMP
my-instance-template-googleapisproxy-vpn2  n1-highcpu-2               2017-08-21T12:33:44
Instance Group for VPN1
$gcloud compute instance-groups managed create my-instance-group-managed-europe-west1-b-d-googleapisproxy --size 1 --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --zone europe-west1-b --template my-instance-template-googleapisproxy-vpn1 --base-instance-name my-googleapisproxy-vm
Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/instanceGroupManagers/my-instance-group-managed-europe-west1-b-d-googleapisproxy].
NAME                                                         LOCATION        SCOPE  BASE_INSTANCE_NAME                    SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                          AUTOSCALED
my-instance-group-managed-europe-west1-b-d-googleapisproxy  europe-west1-b  zone   my-europe-west1-b-d-googleapisproxy  0     1            my-instance-template-googleapisproxy-vpn1  no
Instance Group for VPN2
$gcloud compute instance-groups managed create my-instance-group-managed-europe-west1-c-d-googleapisproxy --size 1 --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --zone europe-west1-c --template my-instance-template-googleapisproxy-vpn2 --base-instance-name my-googleapisproxy-vm
Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/instanceGroupManagers/my-instance-group-managed-europe-west1-c-d-googleapisproxy].
NAME                                                         LOCATION        SCOPE  BASE_INSTANCE_NAME                    SIZE  TARGET_SIZE  INSTANCE_TEMPLATE                         AUTOSCALED
my-instance-group-managed-europe-west1-c-d-googleapisproxy  europe-west1-c  zone   my-europe-west1-c-d-googleapisproxy  0     1            my-instance-template-googleapisproxy-vpn2  no
Backend Service
$gcloud compute backend-services create my-backend-service-googleapisproxy --description "Backend service for load balancing across googleapisproxy instance groups" --health-checks my-healthcheck-googleapisproxy --session-affinity NONE --timeout 5m --protocol TCP --port-name ndl-aas --region europe-west1 --load-balancing-scheme INTERNAL
Created [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/backendServices/my-backend-service-googleapisproxy].
NAME                                  BACKENDS  PROTOCOL
my-backend-service-googleapisproxy            TCP
Add instance group backends
$gcloud compute backend-services add-backend my-backend-service-googleapisproxy --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --instance-group my-instance-group-managed-europe-west1-b-d-googleapisproxy --instance-group-zone europe-west1-b --region europe-west1
Updated [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/backendServices/my-backend-service-googleapisproxy].

$gcloud compute backend-services add-backend my-backend-service-googleapisproxy --description "Layer 3 load balanced instance group: Network proxy for accessing Google APIs" --instance-group my-instance-group-managed-europe-west1-c-d-googleapisproxy --instance-group-zone europe-west1-c --region europe-west1
Updated [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/backendServices/my-backend-service-googleapisproxy].
Forwarding Rule
$gcloud compute forwarding-rules create my-forwarding-rule-googleapisproxy --description "forwarding rule to direct network traffic to the googleapisproxy load balancer" --backend-service my-backend-service-googleapisproxy --address 10.0.0.1 --load-balancing-scheme INTERNAL --ip-protocol TCP --network projects/my-project/global/networks/my-network --subnet projects/my-project/regions/europe-west1/subnetworks/subnet-1 --ports 3128 --region europe-west1
 Created [https://www.googleapis.com/compute/v1/projects/my-project/regions/europe-west1/forwardingRules/my-forwarding-rule-googleapisproxy].
Autoscaling
$gcloud compute instance-groups managed set-autoscaling my-instance-group-managed-europe-west1-c-d-googleapisproxy --description "Autoscaling policy for scaling up and down the number of instances in an instance group for googleapisproxy" --min-num-replicas 1 --max-num-replicas 4 --zone europe-west1-c --scale-based-on-cpu --target-cpu-utilization 0.75 --cool-down-period 180
Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/autoscalers/my-instance-group-managed-europe-west1-c-d-googleapisprox-rsjb].
---
autoscalingPolicy:
  coolDownPeriodSec: 180
  cpuUtilization:
    utilizationTarget: 0.75
  maxNumReplicas: 4
  minNumReplicas: 1
creationTimestamp: '2017-08-21T12:33:44'
description: Autoscaling policy for scaling up and down the number of instances in
  an instance group for googleapisproxy
id: '123456789000000'
kind: compute#autoscaler
name: my-instance-group-managed-europe-west1-c-d-googleapisprox-rsjb
selfLink: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/autoscalers/my-instance-group-managed-europe-west1-c-d-googleapisprox-rsjb
status: ACTIVE
target: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c/instanceGroupManagers/my-instance-group-managed-europe-west1-c-d-googleapisproxy
zone: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-c

$gcloud compute instance-groups managed set-autoscaling my-instance-group-managed-europe-west1-b-d-googleapisproxy --description "Autoscaling policy for scaling up and down the number of instances in an instance group for googleapisproxy" --min-num-replicas 1 --max-num-replicas 3 --zone europe-west1-b --scale-based-on-cpu --target-cpu-utilization 0.75 --cool-down-period 180
Created [https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/autoscalers/my-instance-group-managed-europe-west1-b-d-googleapisprox-9610].
---
autoscalingPolicy:
  coolDownPeriodSec: 180
  cpuUtilization:
    utilizationTarget: 0.75
  maxNumReplicas: 3
  minNumReplicas: 1
creationTimestamp: '2017-08-21T12:33:44'
description: Autoscaling policy for scaling up and down the number of instances in
  an instance group for googleapisproxy
id: '123456789011111'
kind: compute#autoscaler
name: my-instance-group-managed-europe-west1-b-d-googleapisprox-9610
selfLink: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/autoscalers/my-instance-group-managed-europe-west1-b-d-googleapisprox-9610
status: ACTIVE
target: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b/instanceGroupManagers/my-instance-group-managed-europe-west1-b-d-googleapisproxy
zone: https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west1-b