Pub / Sub Messaging Service

Owned by Roger Cruz
Last updated: Aug 31, 2020
4 min read

Intro

Notes on pub/sub service.

Pub/Sub is an asynchronous messaging service that decouples services that produce events from services that process events.

Documentation

Tips and Tidbits

 

  • Topic: A named resource to which messages are sent by publishers. Created in a GCP Project.

  • Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. Create a subscription from Google Console

  • Message: The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscriber

  • Message attribute: A key-value pair that a publisher can define for a message. For example, key iana.org/language_tag and value en could be added to messages to mark them as readable by an English-speaking subscriber.

  • A publisher application creates and sends messages to a topic.

  • Subscriber applications create a subscription to a topic to receive messages from it.

    • Pub/Sub service forwards messages from a topic to all of its subscriptions, individually.

      • E.g Topic C sends Subscription YC and ZC

    • The Pub/Sub service ensures that published messages are retained (stored) on behalf of subscriptions.

      • It is removed from storage once it is acknowledged by any subscriber consuming messages from that subscription.

    • Pull subscribers can also be any application that can make HTTPS requests to pubsub.googleapis.com

    • Push subscribers must be Webhook endpoints that can accept POST requests over HTTPS.

  • Communication can be one-to-many (fan-out), many-to-one (fan-in), and many-to-many.

Source: What Is Pub/Sub?

  • My experience: messages sent to a topic BEFORE a subscription has been added appear to be lost/discarded. I couldn’t see them after adding the subscription.

  • Pub/Sub does not support API keys as an authentication method

  • Access control can be configured at the project level and at the individual resource level (ie topic or subscription).

    • Limited permissions (such as read but can’t delete messages) can be granted.

  • Every Pub/Sub method requires the caller to have the necessary permissions. 

Emulator

There is an emulator for the service so you can’t avoid doing app development in a live environment.

Authentication

 

  • Pub/Sub authentication documentation: https://cloud.google.com/pubsub/docs/authentication

  • Service accounts and user accounts are supported as authentication methods but Service Accounts are recommended for almost all use cases

    • Pub/Sub does not support API keys as an authentication method

      • I don’t understand this comment in the docs because a Fiddler of the Google API Browser shows a key being specified. Is it an API key (Setting up API keys)?

        • Looks like it is a service account key

Access Control

 

  • Access control

  • The Pub/Sub IAM roles with a corresponding list of all the permissions each role includes is found here: Roles

  • For a publisher to be able to write to a Topic, it must have IAM permissions ofpubsub.topics.publish

Creating A Topic

 

The GCP help page on creating a pub/sub via gcloud is here: https://cloud.google.com/sdk/gcloud/reference/pubsub/topics/create

gcloud --project rogercruz pubsub topics create mytopic

 

Using the REST API method: projects.topics.create

  • Type in the path as specifiedin Google API Explorer.

  • OK response indicates topic was created.

  • An HTTPS capture with Fiddler shows the PUT command to the URL path for

 

  • Confirm on Google’s console that you can see the newly created topic

 

 

Permissions

 

If your topic is to ingest data from Google’s logging service, you must allow the sink’s service account to write to the topic by binding the Servive Account with the IAM role roles/pubsub.publisher. See

gcloud pubsub topics add-iam-policy-binding <topic-name> --member serviceAccount:<LOG-SINK-SERVICE-ACCOUNT> --role roles/pubsub.publisher

 

Publish A Message

 

Read this guide on publishing a message and its format: Publishing messages to topics

Using gcloud pubsub topics publish

 

gcloud pubsub topics publish topic-roger --message "Hello World!" --project rogercruz messageIds: - '454651473899131'

 

Publish Events Extracted From GCP Logs

 

A topic can receive events from GCP’s logging services. To do this, you need to create a sink which monitors the logs for the events you desire and specifies the topic as the destination.

Â