#Big Data

Performing joins especially with large datasets will be a huge challenge in data processing. Here is the fix. 👇 1️⃣ Make a broadcast join Instead of shuffling 50TB of data across the network to find matches, you should send a copy of the small table to every single worker node. 2️⃣ Map-Side Operation This converts the operation into a local lookup. Each executor holds the full 100MB table in RAM and joins it against its local slice of the 50TB data. 3️⃣ The Memory Trap Be careful -> if that “small” table grows too big (e.g., 2GB), broadcasting it will cause Out-Of-Memory (OOM) errors on the executors and crash the application. 4️⃣ Configuration Threshold Check the spark.sql.autoBroadcastJoinThreshold. If the table is slightly larger than the default (usually 10MB), the system might default to a slow Sort-Merge join unless I increase this limit. #dataengineering #bigdata #coding 🏷️ Data Engineering, Apache Spark, Coding Interview, Tech Interview, Big Data Processing, Spark, Python

What are the big data use cases you have tried? Comment below 👇 #bigdata #bigdataanalytics #bigdatatechnologies #bigdataanalysis #trendingreels

The best projects serve a real use case Comment “data” for all the links and project descriptions #tech #data #datascience #ml #explore

Why does everybody succeed with BigQuery and you are not? Save $$$ THOUSANDS of dollars with these BigQuery optimizations: (most of them could be applied in your data warehouse as well or use analogy) 🔒 SAVE IT! 1️⃣ analyse query plan execution 2️⃣ avoid SELECT * 3️⃣ cluster your tables 4️⃣ partition your tables 5️⃣ apply filter with partitions on partitioned tables 6️⃣ reduce data before using a JOIN 7️⃣ for multiple joins start with the largest table 8️⃣ apply search index for faster string data search (cool BQ function) What are your favourite cloud data warehouses tricks? #sql #data #bigdata #datascience #dataanalyst #dataanalytics #dataengineer #datascientist #analytics

Data is the new gold. Big data describes large and diverse datasets that are huge in volume and also rapidly grow in size over time. Big data is used in machine learning, predictive modeling, and other advanced analytics to solve business problems and make informed decisions. A data engineer develops, builds, maintains, and manages data pipelines. This requires working with large datasets, databases , and the software used to analyze them – including cloud systems like AWS or Azure. The primary focus of a data engineer is to ensure that data flows smoothly from its source to its destination efficiently and securely. The data engineer is the first line of data ingestion, cleaning and wrangling, and transformation using tools such as Python, PySpark and SQL. #dataengineer #tech #corporategirlie #motivation #fyp

In my first years as a data scientist, I wasted hours on broken SQL, slow pandas scripts, messy Flask deployments, and “works on my machine” chaos. These 4 tools fixed that: • dbt → modular, documented SQL transformations • Polars → faster, cleaner alternative to pandas • FastAPI → quick, reliable model deployment • Docker → consistent environments, no more deployment nightmares If you’re just starting out, learning these early will save you months of frustration.

how I analyze data as a Business Analyst at Spotify! Spotify商業分析師如何分析數據？ ft. @tableausoftware #womenintech #businessanalyst #dataanalyst #gendata #datafam #spotify

Day 3: Importing Data into Power BI (+ importing data from the web!) #dataanalyst #dataanalysis #dataanalytics #powerbi #powerquery

watch this if you want to become a data analyst in 2026, these are my top simple tips 📊 1. Learn SQL: its the tool you’ll use to get data from databases, and then use to analyse business performance 2. Learn Excel or something similar: it’s great for ad hoc analysis and building engaging charts and diagrams 3. Get familiar with a reporting tool, you don’t need to be great at this just an understanding is fine 4. The core skills are communicating your insights clearly and understanding business metrics Save this and come back to it when you’re planning what to learn, I have links on my profile for courses/guides for each of these aspects!

Data analysis isn't about crunching numbers anymore 🤯 Comment "docs" for a list of prompts and a guide around data analysis with AI. Mastering AI for data visualization is less about being a technician and more about becoming a strategic storyteller. This single shift in focus is what separates junior analysts from senior leaders.

Let’s work on an Exploratory Data Analysis together in SQL In this analysis, we’re looking at social media vs. productivity data. The dataset is from Kaggle, and it looks to be a synthetic dataset. But either way, it’s a good dataset to practice EDAs Typically for EDAs, I like to look for 3 things: - Distributions - Relationships - Outliers We covered the first 2 in this video. Comment below if this was helpful, and I can make more of these!! #exploratorydataanalysis #eda #sql #dataanalytics #datascience

Data Analytics Road map (6-9 months) https://drive.google.com/drive/folders/17KOCp6F1JGqOCwIdryzcDykNCSu93Ltc?usp=sharing Built from my personal interview experiences(Interviews given - 5+) Duration - 1-2 Months - Basics Learn basic - intermediate SQL(joins) from youtube/udemy Basic Python from youtube/udemy/Leetcode Basic Excel Duration 2-3 Months - Intermediate Practice intermediate to advanced SQL on Data Lemur/Leetcode/WiseOwl Practice easy-intermediate python questions on Leetcode/Hackerrank Start BI - Power BI tutorial from youtube/udemy Duration 3-4 Months - Advanced Learn Pandas/pyspark, practice EDA on csv files from Kaggle datasets on jupyter notebook/colab Practice advanced SQL questions(window functions) Build BI projects from kaggle datasets/Datacamp Github profile to showcase your projects + LinkedIn Theoretical knowledge on ETL pipelines/ Data warehousing concepts(Chat GPT) Resources SQL - Theory - W3Schools(free)/Udemy(paid), Practice - Leetcode/Data Lemur Python - Theory - Youtube/Udemy, Practice - Leetcode(easy to medium) Data Warehousing+ETL - Tutorials Point/Udemy, Datacamp/Chat GPT Power BI/Tableau - Datacamp, wiseowl Pandas/Pyspark - Datacamp, Leetcode, Kaggle Basic Excel . . . . . . #big4 #fyp #data #analytics #ootd #grwm