Deactivate Data Pipeline Architecture
There are mainly 13 steps on this pipeline. The task of every step is as follows.
Read all JSON files of the previous hour from GCP cloud storage.
JSON to Object:
This step just converts JSON to a Python dictionary for further execution.
This step checks the hash, is the hash of a valid project or not. If the data is valid then send it to the next step otherwise skip it.
Filter Custom Fields:
This step validates the custom fields of a project. If it found any valid custom fields then it creates an array of those custom fields. Send the array with other data to the next step.
Clean and Cast Data:
Create a fresh array of data. This step cast data type for the necessary fields and just take the valid fields and skip unnecessary fields. After this step, we get a fresh and clean array of data.
Request for mail integrations:
This checks the project on MongoDB. If the mail integration (MailChimp, MailJet, WeMail) is true then send a request to the appsero-api server for sending the mail to the provider list. If there is no mail integration then it doesn’t send any request.
Insert Sites Into MongoDB:
Insert the site into MongoDB if this is a new site. If the site already exists then it updates all information on the site. MongoDB database is used to store the latest data for each site.
Convert Custom Fields to JSON for BigQuery:
We are storing custom fields as a JSON string in BigQuery. This step converts the custom fields array to JSON string so that this can be inserted into BigQuery.
Insert Sites Into BigQuery:
Insert the site into BigQuery. We are using BigQuery as a data warehouse. We store all historical data of sites into BigQuery.
Validate Reason ID:
This step checks the reason id on MongoDB. Here reason_id is the slug of the deactivation reason. If the reason_id is valid then it sends the data to the next step otherwise skip the data.
Skip Unnecessary Data:
This is step clean data for the deactivation table. This step removes unnecessary fields for the elements.
Store Deactivation to MongoDB:
This step saves the latest deactivation to the MongoDB database. MongoDB database doesn’t contain historical data, deactivation just contains the latest deactivation of a site.
Store Deactivation to BigQuery:
This step saves all deactivations to the BigQuery. BigQuery stores historical data of deactivations.