Skip to main content

Python Code Snippets

 List Buckets from Python

--Requirement google.cloud ,google.cloud.storage, protobuff
import os
from google.cloud import storage
os.environ['GOOGLE_APPLICATION_CREDENTIALS']=r"C:\service_account.json"

#instantiates a client
sotrage_client=storage.Client()

#lists all the available buckets
for bucket in storage_client.list_buckets():
print(bucket)
bucket_iterator=bucket_list_blobs()
for resource in bucket_iterator:
print(resource)


 Bigquery from Python

--Requirement google.cloud ,google.cloud.bigquery, protobuff
import os
from google.cloud import storage
os.environ['GOOGLE_APPLICATION_CREDENTIALS']=r"C:\service_account.json"
project_id='<>'
import google.auth
from google.cloud import bigquery
client=bigquery.Client()
#client=bigquery.Client(credentials=crendentials_file,project=project_id,)

query='''SELECT * FROM dataset.tableNM Limit 2;'''

query_job=client.query(query) # makes an API Request to Google Bigquery

print("Query Data is")
for row in query_job:
print(row) #Row Values can be accessed by Field Name or Index

#Need to have bigquery.jobs.create permission enabled to project id
#can get credentials directly from GUI
#import google.auth
#from google.cloud import bigquery
#credentials,project_id=google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])


Comments

Popular posts from this blog

LookML

  What Is LookML? LookML is a language for describing dimensions, aggregates, calculations, and data relationships in a SQL database. Looker uses a model written in LookML to construct SQL queries against a particular database. LookML Projects A LookML Project is a collection of model, view, and dashboard files that are typically version controlled together via a Git repository. The model files contain information about which tables to use, and how they should be joined together. The view files contain information about how to calculate information about each table (or across multiple tables if the joins permit them). LookML separates structure from content, so the query structure (how tables are joined) is independent of the query content (the columns to access, derived fields, aggregate functions to compute, and filtering expressions to apply). LookML separates content of queries from structure of queries SQL Queries Generated by Looker For data analysts, LookML fosters DRY style...

A Deep Dive Into Google BigQuery Architecture

Introduction Google’s BigQuery is an enterprise-grade cloud-native data warehouse. BigQuery was first launched as a service in 2010 with general availability in November 2011. Since inception, BigQuery has evolved into a more economical and fully-managed data warehouse which can run blazing fast interactive and ad-hoc queries on datasets of petabyte-scale. In addition, BigQuery now integrates with a variety of Google Cloud Platform (GCP) services and third-party tools which makes it more useful. BigQuery is serverless, or more precisely data warehouse as a service. There are no servers to manage or database software to install. BigQuery service manages underlying software as well as infrastructure including scalability and high-availability. The pricing model is quite simple - for every 1 TB of data processed you pay $5. BigQuery exposes simple client interface which enables users to run interactive queries. Overall, you don’t need to know much about underlying BigQuery architecture or...

Python - Working With Conda

 Python - Working with Conda Getting Conda Environment Info $conda info Login to Conda Environment $export PATH=”/app/miniconda3/bin:$PATH” $source activate [env_name] How to get Module installed path >>>import module >>>print(module.__file__)   Export Modules to file $conda list --explicit > env_name.txt Listing Environments $coda env list Creating New Environment $conda config --add channels conda-forge   #after this we can get file ~/.condarc $vi condarc #set proxy in condarc file ~/.condarc channels: -           conda-forge -           defaults proxy-servers:       http:http://company.com       https:http://company.com   $conda create --name [env_name] --f...