A Journey to Data Architect ..

Connection ID Connection Type Host Port airflow_db mysql airflow-sqlproxy-service.default.svc.cluster.local 3306 beeline_default beeline localhost 10000 cassandra_default cassandra cassandra 9042 druid_broker_default druid druid-broker 8082 druid_ingest_default druid druid-overlord 8081 hiveserver2_default hiveserver2 localhost 10000 http_default http https://www.httpbin.org metastore_default hive_metastore localhost 9083 mongo_default mongo mongo 27017 mysql_default mysql mysql 1433 presto_default presto localhost 3400 redis_default redis redis 6379 sftp_default sftp localhost 22 vertica_default vertica localh...

Availability,durability and scalability are design characteristics you need to consider when developing a cloud application.  Durability ensures that your data is protected and available. Snapshots are a viable way of backing your data. Durability is a measurement related to data protection. Autoscaling is a method for implementing scalable applications. Health checks and Managed Instance groups improve availability. Availability is a measurement of uptime

 Continuous integration/continuous delivery (CI/CD) pipelines automate the testing and delivery of code by monitoring a controlled software repository. When new software is checked in, the pipeline orchestrator will first run unit tests. If the tests are successful , a deployment package is built and saved to a Container Registry,which completes the steps for continuous integration.Continuous delivery deploys your images or artifacts to the operational environment you specify, such as App Engine or GKE. With Google Cloud, you can use Cloud Source Repositories as your Version Control repository. Unit testing and container artifacts can be produced through Cloud Build and automated through build triggers. Common places to save and manage your finalized images include Container Registry and Artifact Registry. Continuous Delivery can also be implemented through steps specified in Cloud Build and implementing build triggers to apply build operations when a new manifest is added to a mon...

 git clone existing repo in IDE  git clone https://github.com/org/repo.git make necessary changes in the files  git remote -v it list all the remote urls in the local system IDE origin https://github.com/org/repo.git (fetch) origin https://github.com/org/repo.git (push) git rename existing origin url to new old-orign git remote rename origin old-origin git add new repo url to origin git remote add origin https://github.com/org/repo2.git verify origin urls git remote -v should get 4 urls 2 for old-origin(old repo) and 2 for (new repo) origin  git reinitialize  git init .   Reinitialized existing Git repository in C:/repo/.git/ git add all files git add . commit the changes  git commit -m "Initial" git remove .idea files  git rm -r --cached .idea  removes already tracked files in git rm -r .idea  this deletes all .idea files including the folder locally git push -u origin --all git remote remove old-origin

Spanner  from google.cloud import spanner import google.auth from google.auth import impersonated_credentials credentials ,project = google.auth.default() #get default credentials and project target_credentials = impersonated_credentials.Credentials(source_credentials=credentials,  target_principals = impersonate_sa_to, #give sa to which you want to impersonate  target_scopes=["https://www.googleapis.com/auth/cloud-platform"] ) spanner_client=spanner.Client(credentials=target_credentials) instance = spanner_client.instance(instance_id) database=instance.database(database_id)

 Setting up of custom python Environment in Notebook  Refer  RStudio  for using virutalenv Below is the process to setup using conda  from a terminal in Jupiter ,create and activate a new conda environment conda create -n <env-name> conda activate <env-name> (Switch to bash to enable conda) in that new activated environment , install package ipykernel conda install ipykernal Register conda environment python -m ipykernal install --name python-env --display-name "Conda Python Env" --user Refresh the launcher to see new kernel which you created in above process

Command:  hadoop distcp -libjars $jar1 --files $file1 -Dgoogle.cloud.auth.service.account.json.keyfile=SERVICE_ACCOUNT.json -Dfs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS -m 1 hdfs://${hadoop_loc}  gs://${gcs_loc}