A Journey to Data Architect ..

 If you want to use your own or nexus repo to download the packages , use below process Create folder if not exists in C:\Users\username\AppData\Roaming\pip in windows Create pip.ini file on windows AppData>Roaming>pip folder  Copy the below lines and place in pip.ini file                [global]               index-url=https://<repo.com>/nexus/repository/ae-pypi-group/simple               trusted-host=<repo.com> On unix default configuration file is $HOME/.config/pip/pip.conf which respects XDG_CONFIG_HOME environment variable. On MacOs the configuration file is $HOME/Library/Application Support/pip/pip.conf

HTML Table Generator Command Description Comments gcloud config unset auth/impoersonate_service_account  Unset Impersonate SA    gcloud config set auth/impersonate_service_account sa   Set Impersonate SA    gcloud config list  Gcloud Conifgutation list    bq query --project_id={project_id} --format=csv --use_legacy_sql=false 'select * from tbl'   Run SQL Query   

 Creating New Conda Environments.. Create New Environment Activate Virtual Environment Install Packages if any Install Jupyter Notebook if needed to add this new environment as a kernel into your Jupyter Notebook. $conda create -n env-name python=3.8 numpy pandas -c https://repo.anaconda.com/pkgs/snowflake $conda activate env-name $conda install snowflake-snowpack-python $conda activate env-name pip install ipykernel python -m ipykernel install --name env-name

Connection ID Connection Type Host Port airflow_db mysql airflow-sqlproxy-service.default.svc.cluster.local 3306 beeline_default beeline localhost 10000 cassandra_default cassandra cassandra 9042 druid_broker_default druid druid-broker 8082 druid_ingest_default druid druid-overlord 8081 hiveserver2_default hiveserver2 localhost 10000 http_default http https://www.httpbin.org metastore_default hive_metastore localhost 9083 mongo_default mongo mongo 27017 mysql_default mysql mysql 1433 presto_default presto localhost 3400 redis_default redis redis 6379 sftp_default sftp localhost 22 vertica_default vertica localh...

Availability,durability and scalability are design characteristics you need to consider when developing a cloud application.  Durability ensures that your data is protected and available. Snapshots are a viable way of backing your data. Durability is a measurement related to data protection. Autoscaling is a method for implementing scalable applications. Health checks and Managed Instance groups improve availability. Availability is a measurement of uptime

 Continuous integration/continuous delivery (CI/CD) pipelines automate the testing and delivery of code by monitoring a controlled software repository. When new software is checked in, the pipeline orchestrator will first run unit tests. If the tests are successful , a deployment package is built and saved to a Container Registry,which completes the steps for continuous integration.Continuous delivery deploys your images or artifacts to the operational environment you specify, such as App Engine or GKE. With Google Cloud, you can use Cloud Source Repositories as your Version Control repository. Unit testing and container artifacts can be produced through Cloud Build and automated through build triggers. Common places to save and manage your finalized images include Container Registry and Artifact Registry. Continuous Delivery can also be implemented through steps specified in Cloud Build and implementing build triggers to apply build operations when a new manifest is added to a mon...

 git clone existing repo in IDE  git clone https://github.com/org/repo.git make necessary changes in the files  git remote -v it list all the remote urls in the local system IDE origin https://github.com/org/repo.git (fetch) origin https://github.com/org/repo.git (push) git rename existing origin url to new old-orign git remote rename origin old-origin git add new repo url to origin git remote add origin https://github.com/org/repo2.git verify origin urls git remote -v should get 4 urls 2 for old-origin(old repo) and 2 for (new repo) origin  git reinitialize  git init .   Reinitialized existing Git repository in C:/repo/.git/ git add all files git add . commit the changes  git commit -m "Initial" git remove .idea files  git rm -r --cached .idea  removes already tracked files in git rm -r .idea  this deletes all .idea files including the folder locally git push -u origin --all git remote remove old-origin