Jul 19, 2022Python | Poetry | Dockerfile and private repositoriesWhen we have an ecosystem like python — poetry — dockerfile and private repositories, there are few areas that will cause issues. Here are my notes while solving such issues. Private repo credential management Scenario We have a common python library (let’s call it lib-1 )in a private git repository We have another repository…Python2 min readPython2 min read
Jun 18, 2022gitlab CI pipeline — learnings, tips and tricksIn this article, we will cover few areas to make our CICD pipelines more efficient. Understanding Gitlab CI CD rules Gitlab CICD rules are a little confusing at the beginning and we can easily make many mistakes. The below video helped me in understanding them to effectively build pipelines. Rule evaluation will stop when first…3 min read3 min read
Jun 10, 2022Open source terminal window screen capture tools for developersDevelopers need to radiate complex steps and terminal commands to other developers. There are many open source tools to help us in this area. But it is hard to search them and narrow them down. In this article, I am trying to consolidate my favourite picks. Following tools helps developers…Developer2 min readDeveloper2 min read
Apr 27, 2022Setup mongo using Kubernetes community operator in local dev boxSetting up mongo in local dev box is little tricky. The official documents are good place to start. But I faced issues with getting them all working in a local setup for development purpose. Setup mongo operator helm repo add mongodb https://mongodb.github.io/helm-charts helm install community-operator mongodb/community-operatorSetup mongo database resource Let’s setup mongo database…Mongodb2 min readMongodb2 min read
Feb 18, 2022Terraform — Best practices and project setupGetting comfortable with terraform is a huge benefit while working with cloud providers. While working in a team, often if team members start to modify the cloud resources, let’s say, AWS resources, directly in the AWS console, it gets harder to track the changes, reason for the change etc., In…Data Engineering3 min readData Engineering3 min read
Dec 30, 2021Visualize parquet files with Apache Superset using Trino or PrestoSQLMany times, I like to visualize contents in formats like parquet, csv, json etc in Apache Superset. This article tries to provide a demo setup to achieve this. Tech stack Trino (formerly PrestoSQL) Minio — for hosting the file. AWS S3 compatible. Hive Metastore — for accessing files from Trino using Hive…Presto3 min readPresto3 min read
May 14, 2021DB data type — Varchar vs Text dilemmaOn a quick look, it feels like varchar(n) is better, because it feels like we are saving disk space by specifying exactly what we are going to use. Is it really worth it ? Why can’t we keep it simple and call it text data type ? Postgres Character types More details info…Database2 min readDatabase2 min read
Apr 12, 2021Java developer setup for learning and experimentsWorking with multiple Java versions As of now, there are still applications that work only with JDK 8. There are other applications that work with JDK 11 max. There are few which work on latest & greatest Java 16. Having a local setup to easily switch between them will be very handy. The following steps…Java2 min readJava2 min read
Mar 26, 2021Mindset of a technical interviewerAs with many professionals, I have been on both sides — interviewer & interviewee. Tried to capture what I like to practice when I interview someone. Though the concepts discussed here are generic, some of my thoughts may be biased towards software programmer interview. An organisation is a soil — every candidate is a seed There are different kinds of soil…Hiring6 min readHiring6 min read
Mar 10, 2021Apache Spark — local setup for efficient development and debuggingHaving a good local setup is always handy to deploy spark apps, develop, debug and learn the spark ecosystem. Local standalone spark cluster ./sbin/start-master.sh ./sbin/start-worker.sh spark://<system_name>:7077 # Once done, we could stop all of them ./sbin/stop-all.sh Having a master and worker based local setup is handy to simulate different needs & life cycle of…Pyspark4 min readPyspark4 min read
