Tuesday, September 12, 2023

How ChatGPT Works

 https://mitsloanedtech.mit.edu/ai/basics/how-chatgpt-works-a-non-technical-primer/#Watch_the_Video

Monday, August 21, 2023

Wednesday, June 21, 2023

Aman Shark Tank

 https://youtube.com/clip/UgkxS9MzQ5ZXe8yinaft11ZJOaUvtHv9rqJ9

 

https://www.youtube.com/watch?v=En5uGDtL-MQ

start time 25m 35s

 https://www.youtube.com/watch?v=ddfQLld0uLU

start time : 19m 25s


Saturday, January 28, 2023

DynamoDB should i use scan or query ?

Difference Between Query and Scan in DynamoDB

Query and Scan are two operations available in DynamoDB SDK and CLI for fetching a collection of items. While they might seem to serve a similar purpose, the difference between them is vital. While Scan is "scanning" through the whole table looking for elements matching criteria, Query is performing a direct lookup to a selected partition based on primary or secondary partition/hash key. You will also see the difference in speed. While Query usually returns results within 100ms, Scan might even take a few hours to find the relevant piece of data.

When querying DynamoDB, When You Should Use Scan vs Query ?
Generally speaking, you should always favor Query over Scan. When it's not possible (for example, when you're looking for a piece of data with a key that is unknown to you), and if it's a frequently used pattern, consider adding a GSI (Global Secondary Index) to index that attribute and enable Query. In the last resort, use Scan.
I need to use DynamoDB scan. Can I make it faster?

You can consider using DynamoDB Parallel Scan. It uses multiple threads to run multiple scans at once scanning multiple parts of your table space simultaneously.

When To Use DynamoDB Scan

But sometimes using scans is inevitable, you only need to use them sparingly and with knowledge of the consequences. Here are use-cases by scans might make sense: 
  • Getting all the items from the table because you want to remove or migrate them
  • If your table is really small (< 10 MB) and it takes a few calls to scan through all of it
  • Scan operation can be also used on secondary index, especially if we are dealing with sparse index. In this case, scan will return only items that have an attribute which is indexed by the selected index, e.g. deleted entities.

Restricting usage of DynamoDB Scans using AWS IAM
If you don't want to rely on "good will" and want to codify the best practices instead, you can restrict usage of Scans at all for all users in your organization/AWS account using following IAM policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "dynamodb:Scan", "Resource": "*" } ] }
DynamoDB Scan Filtering 
In order to minimize the amount of data returned from the database to the application, you can use FilterExpressions. Filter expressions allow you to select and return only a subset of scan results. 

How fast is DynamoDB scan?
DynamoDB Scan is not a fast operation. Because it goes through the whole table to look for the data, it has O(n) computational complexity. If you need to fetch data fast, use Query or Get operations instead. 

What is the DynamoDB scan cost?
DynamoDB Scan cost depends on the amount of data it scans, not the amount of data it returns. 

Parallel Scan in DynamoDB Scans are generally speaking slow. To make that process faster, you can use a feature called "Parallel Scans" which divide the whole DynamoDB Table into Segments. A separate thread/worker then processes each Segment so N workers can work simultaneously to go through the whole keyspace faster. Creating Parallel Scan is quite easy. Each of your workers, when issuing a Scan request should include two additional parameters: Segment - Number of segments to be scanned by a particular worker Total Segments - Total amount of Segments/Workers/Threads
But, be careful with Parallel scans as they can drain your provisioned read capacity pretty quickly incurring high costs and degrading the performance of your table.

Sunday, August 28, 2022

gRPC vs REST vs GraphQL

𝗚𝗿𝗮𝗽𝗵𝗹𝗤𝗟 is a data query language that uniquely allows clients to request any specific data that they need. As opposed to REST’s HTTP methods, GraphQL uses queries, mutations, and subscriptions for sourcing and manipulating data. Queries request data from the server while mutations send data to and modify data gated by the server. Subscriptions get live updates when data changes, usually through Websockets. 𝗥𝗘𝗦𝗧 The most popular and most used API format on the list is REST, which stands for representational state transfer. REST APIs use HTTP methods like GET, POST, PUT, and DELETE to access and manipulate data within uniform resource identifiers (URIs). 𝗴𝗥𝗣𝗖 is a high-performance, open source remote procedural call (RPC) framework created by Google. In gRPC, protocol buffers make API requests to the server. These API requests and responses are protocol buffer messages, and each one must have its own type defined using the protocol buffers language. So, 𝘄𝗵𝗶𝗰𝗵 𝗼𝗻𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗯𝗲𝘀𝘁? - The winner is, of course, “𝗶𝘁 𝗱𝗲𝗽𝗲𝗻𝗱𝘀.” Each technology has strong benefits, but those come with trade-offs. Your needs may best be served by a combination of technologies. - 𝗥𝗘𝗦𝗧, 𝗼𝗿 𝗮𝘁 𝗹𝗲𝗮𝘀𝘁 𝗝𝗦𝗢𝗡 𝗼𝘃𝗲𝗿 𝗛𝗧𝗧𝗣, is the most ubiquitous standard for web-based APIs. This means it’s easy to get started, you can use a wide variety of languages, and it works natively with web browsers. - In their own way, 𝗚𝗿𝗮𝗽𝗵𝗤𝗟 𝗮𝗻𝗱 𝗴𝗥𝗣𝗖 address some of the limitations of REST. - 𝗚𝗿𝗮𝗽𝗵𝗤𝗟 allows a client to specify just the information they need, which can greatly reduce duplicate or unnecessary data being transmitted. But, it requires additional setup and training. - 𝗴𝗥𝗣𝗖 is built for fast transport, leveraging HTTP/2. This requires a well known contract, typically using Protocol Buffers, that is shared by the client and server.

Wednesday, January 23, 2019

REST APIs and DateTime Format

Found some interesting Articles

http://apiux.com/2013/03/20/5-laws-api-dates-and-times/ 

https://www.moesif.com/blog/technical/timestamp/manage-datetime-timestamp-timezones-in-api/
prefer ISO 8601 (yyyy-MM-dd'T'HH:mm:ssZ) because it’s human readable and has a specified timezone. There is no ambiguity if the epoch is in seconds or milliseconds. ISO 8601 strings are encoded in a way that enable decent string sorting and comparison.

Epoch has benefits over ISO 8601 also. If you’re dealing with high volume sensor data or APIs for IoT devices, you may want epoch since it enables smaller payloads without accounting for compression and more familiar with those coming from an IoT/embedded background

and
https://stackoverflow.com/questions/9581692/recommended-date-format-for-rest-get-api

Sunday, December 31, 2017

Docker Stack

What is Docker Stack ?
It is a collection of services that make up an application in a specific environment.
A stack file is a file in YAML format similar to docker-compose.yml , that defines one or more services

Why should we use it ?
  • convenient way to automatically deploy multiple services that are linked to each other.
  • Stack files define env variables , deployment tags , the number of containers and related environment specific configuration
  • Because of this you should use seperate stack file for dev , stage and production 

 sample command looks like



The Difference
Docker Compose is a Python project. Originally, there was a Python project known as fig which was used to parse fig.yml files to bring up stacks of Docker containers. Everybody loved it, especially the Docker folks, so it got reincarnated as docker compose to be closer to the product. But it was still in Python, operating on top of the Docker Engine.

You still have to install docker-compose separately to use it with Docker on your machine.

The Docker Stack functionality, is included with the Docker engine. You don’t need to install additional packages to use it Deploying docker stacks is part of the swarm mode. It supports the same kinds of compose files, but the handling happens in Go code, inside of the Docker Engine. You also have to create a one-machine “swarm” before being able to use the stack command, but that’ not a big issue.

Docker stack does not support docker-compose.yml files which are written according to the version 2 specification. It has to be the most recent one, which is 3.

The conclusion
As docker stack does everything docker compose does, it’s a safe bet that docker stack will prevail. This means that docker-compose will probably be deprecated and won’t be supported eventually.

However, switching your workflows to using docker stack is neither hard nor much overhead for most users. You can do it while upgrading your docker compose files from version 2 to 3 with comparably low effort.

If you’re new to the Docker world, or are choosing the technology to use for a new project - by all means, stick to using docker stack deploy.