On a quick look, it feels like varchar(n) is better, because it feels like we are saving disk space by specifying exactly what we are going to use. Is it really worth it ? Why can’t we keep it simple and call it
text data type ?
More details info — postgres docs
As of now, there are still applications that work only with JDK 8. There are other applications that work with JDK 11 max. There are few which work on latest & greatest Java 16. Having a local setup to easily switch between them will be very handy.
The following steps are indented for Linux & Mac. I haven’t tried this on Windows for this problem.
jEnv — is an amazing command line tool that can help us.
$ git clone https://github.com/jenv/jenv.git ~/.jenvor$ brew install jenv
As with many professionals, I have been on both sides — interviewer & interviewee. Tried to capture what I like to practice when I interview someone. Though the concepts discussed here are generic, some of my thoughts may be biased towards software programmer interview.
Having a good local setup is always handy to deploy spark apps, develop, debug and learn the spark ecosystem.
./sbin/start-worker.sh spark://<system_name>:7077# Once done, we could stop all of them
While creating spark session, it would be useful to enable event log. This helps in viewing Jobs, stages and DAG after the application is completed.
spark = (…
For data driven applications, choosing the right database solves half the problem. By crystallising data persistence layer first, it helps us ask the right question well before and tends to positively influence other layers. It also reduces unnecessary complexity in the implementation.
In this article, though the right choice ‘depends’ on the problem, I have tried to call out my preferred choice in general on specific categories like RDBMS, Key-Value pair etc.,
In asynchronous server side flows, we like to keep the client informed about the progress or change in state. Many a times, we may choose to go with polling mechanisms. But it has it’s own disadvantages.
Let’s discuss an alternative approach —
Server-sent events and how we can implement this with
As an example, let’s take a look into fastapi based backend endpoint. Here we have a status endpoint which sends server events.
FastAPI is based on starlette. To help us in this flow, we will depend on a small library called sse_starlette
pip install sse-starlette
FastAPI is a promising framework to build high performance web applications that needs Async support. It also brings in best practices for a high quality production application.
Let’s take a quick view of few building blocks that we will use in this article.
FastAPI is build using two important framework for well defined responsibilities :
Starlette is a lightweight ASGI framework/toolkit, which is ideal for building high performance asyncio services.
pydantic enforces type hints…
In micro service architecture., there are times., we would like to execute a command on another service — which runs as a docker container. During these times, it is useful to mount the host docker and delegate the command to host, host executes the command on desired container. To be precise, here we are discussing
Docker outside of Docker (DooD) and not Docker in Docker (DinD)
This is only a sample use case. The core concept is applicable for any docker to docker communication that happens in a typical microservice architecture.
Let’s say we have an Apache Airflow running as…
Apache workflow is designed for complex workflows that are well defined upfront. Creating dynamic workflows in Airflow is a different kind of challenge and there are limited approaches to get it done.
Let’s say we want to create new DAG dynamically where the content of the DAG like what kind of tasks to perform (Operators), schedule of the workflow etc., depends on trigger event.
There are ways to get a dynamic workflow. Couple of options:
for i in range(10)…
Many times, AWS resources are under utilized and each team has it’s own way of managing this. Here we will see how we can stop an idle resource and start it back when we need it.
To achieve this we will be using:
Create SNS topic. We need this for the next step while publishing message from Jenkins. Also this topic will be used when attaching triggers for lambda.
Install Jenkins plugin — Pipeline: AWS Steps.
In the above…