Sep 10Clickhouse database — backup and restore strategyClickhouse provides different ways to take backup and restore. But I faced issues while trying to implement them in a kubernetes based self managed clickhouse. Backup — just schema BACKUP DATABASE default ON CLUSTER 'cluster-name' TO S3( 'https://bucket-name.s3.amazonaws.com/clickhouse_backup/backup-2023_08_24' ) SETTINGS structure_only=true; Above SQL command takes a backup of schema alone…Clickhouse2 min readClickhouse2 min read
Sep 9Clickhouse — sharding and replicationClickhouse is a powerful OLAP database. Coming from a distributed architecture world, we always think of sharding and replication when considering tools and applications. With respect to clickhouse, we need to be extra careful while doing sharding and replication. Prefer scale up over scale out for clickhouse In general, horizontal scaling is the preferred way to architect large…Clickhouse6 min readClickhouse6 min read
Sep 8Clickhouse — useful queries —SQL snippetsWhile using clickhouse as a OLAP database, at times, we spend lot of time in going through the official documentation to find the right function that suite our needs. We may have something in mind but translating that to how it’s done in Clickhouse may be time consuming. Here we…Clickhouse2 min readClickhouse2 min read
Aug 11Github actions for a python projectHere are some code snippets that can help python developers and who use github action as the continuous integration tool. Pytest — Unit test github actionGithub Actions1 min readGithub Actions1 min read
Jul 6Why I will not use Github Actions in futureGithub Actions, as a concept, is very powerful. Tight integration with Github gives it an edge over any other continuous integration & delivery tools. However atleast for me, it is a pain to live with. Half done features There are many conceptual features which say rapid development from the Github side, like reusable…3 min read3 min read
Jun 24Best practices for building docker containerHere I am trying to document different best practices that I like to keep it handy while building docker containers. Use secrets instead of build-arg While handling sensitive data, let’s use --secret provided by latest `BuildKit` build system export MYSECRET=secretpassword docker build --secret id=MYSECRET . Using secret inside docker: FROM ... COPY build-script.sh . RUN --mount=type=secret,id=MYSECRET ./build-script.sh…3 min read3 min read
Jun 6Installing Clickhouse on a kubernetes clusterClickhouse is the fastest and most resource efficient open-source database for real-time apps and analytics. Installing clickhouse on a kubernetes cluster is not straight forward at the moment. This article should help to do this. Kubernetes operators Clickhouse installation on kubernetes may start simple and slowly it would get more & more…3 min read3 min read
Jul 19, 2022Python | Poetry | Dockerfile and private repositoriesWhen we have an ecosystem like python — poetry — dockerfile and private repositories, there are few areas that will cause issues. Here are my notes while solving such issues. Private repo credential management Scenario We have a common python library (let’s call it lib-1 )in a private git repository We have another repository…Python2 min readPython2 min read
Jun 18, 2022gitlab CI pipeline — learnings, tips and tricksIn this article, we will cover few areas to make our CICD pipelines more efficient. Understanding Gitlab CI CD rules Gitlab CICD rules are a little confusing at the beginning and we can easily make many mistakes. The below video helped me in understanding them to effectively build pipelines. Rule evaluation will stop when first…3 min read3 min read
Jun 10, 2022Open source terminal window screen capture tools for developersDevelopers need to radiate complex steps and terminal commands to other developers. There are many open source tools to help us in this area. But it is hard to search them and narrow them down. In this article, I am trying to consolidate my favourite picks. Following tools helps developers…Developer2 min readDeveloper2 min read
