🚀 Data Processing
Data Processing in Python – A Practical Guide (With Examples)
Data Processing is the heart of every data project. Whether you’re working on machine learning, analytics, automation, or building dashboards — sabse pehla step hota hai raw data ko useful format me convert karna.
In this blog, we will understand:
✔ File Handling (Text, TSV, JSON)
✔ Scaling
✔ Encoding
✔ Normalization
✔ Real Python + Pandas + Scikit-learn examples
✔ Easy explanations
⭐ 1. What is Data Processing?
Data Processing means converting raw data into a more meaningful and structured format.
It includes:
Collecting data
Cleaning data
Wrangling and transforming
Analysing
Getting insights
Good processing = Better models + Better decisions
📁 2. Data Wrangling & File Handling in Python
Python provides built-in functions like open(), and powerful libraries like Pandas and JSON to handle files.
📝 Text Files (.txt)
Introduction
A text file is a simple .txt file that stores human-readable characters.
✔ No strict format
✔ Perfect for logs, notes, line-based data
✔ Easy to read/write using open()
✏ Writing to a Text File
with open('example_test.txt', 'w') as file:
file.write("This is sample file for text.\n")
print("File created successfully!")
Writing multiple lines
with open('example_test.txt', 'w') as file:
for i in range(1, 6):
file.write(f"This is line {i} text.\n")
Writing a list of lines
lines = [
'This is First line.',
'This is Second line.',
'This is Third line.',
'This is Fourth line.',
'This is Fifth line.',
]
with open('example_test2.txt', 'w') as file:
for li in lines:
file.write(li + '\n')
📖 Reading a Text File
Using read()
read() gives entire content as a single string.
with open('example_test.txt', 'r') as file:
data1 = file.read()
print(data1)
Using readlines()
Returns list of lines.
with open('example_test.txt', 'r') as file:
data2 = file.readlines()
print(data2)
for line in data2:
print(line.strip())
Using readline() (line-by-line)
file = open("example_test.txt", "r")
print(file.readline())
print(file.readline())
file.close()
Why use with open()?
Because Python automatically calls file.close() for you.
with open("example_test.txt", "r") as file:
print(file.read())
📄 3. Writing & Reading TSV Files (.tsv)
TSV = Tab Separated Values
Used when comma-based CSV causes issues.
Create a TSV
import pandas as pd
data = {
'Emp_ID': [101, 102, 103, 104],
'Name': ['Rahul Sharma', 'Priya Patel', 'Amit Singh', 'Neeta Rao'],
'Department': ['HR', 'Finance', 'IT', 'Marketing'],
'Salary (₹)': [50000, 65000, 80000, 55000],
'Joining_Date': ['2020-01-15', '2019-05-22', '2021-11-10', '2022-03-05'],
'City': ['Mumbai', 'Delhi', 'Bangalore', 'Hyderabad']
}
df = pd.DataFrame(data)
df.to_csv('tab_example.tsv', sep='\t', index=False)
Read a TSV
df_read = pd.read_csv('tab_example.tsv', sep='\t')
df_read
📦 4. Working with JSON Files
JSON is widely used in APIs, configurations, and structured data.
Create JSON File
import json
data = {
"employees": [
{"name": "John", "age": 30, "department": "Sales"},
{"name": "Meera", "age": 25, "department": "Marketing"},
{"name": "Rahul", "age": 35, "department": "HR"}
]
}
with open('json_example.json', 'w') as file:
json.dump(data, file, indent=4)
Read JSON File
with open('json_example.json', 'r') as file:
data_json = json.load(file)
print("Employee information:")
for em in data_json['employees']:
print(f"Name: {em['name']}, Age: {em['age']}, Department: {em['department']}")
⚙️ 5. Data Pre-Processing
Scaling • Encoding • Normalization
Used to prepare data before training ML models.
📏 Feature Scaling
Used because different features have different units.
Example: Salary (₹) vs Age (years)
Machine learning model will give more importance to bigger numbers — salary will dominate.
5.1 StandardScaler (Z-score scaling)
from sklearn.preprocessing import StandardScaler
numeric_data = pd.DataFrame({
'Salary': [30000, 45000, 60000, 80000, 120000],
'Age': [25, 32, 47, 51, 62]
})
std_scaler = StandardScaler()
scaled_std = std_scaler.fit_transform(numeric_data)
df_std = pd.DataFrame(scaled_std, columns=numeric_data.columns)
df_std
Formula:
z = (x - mean) / std
5.2 Min-Max Scaling (0-1 scaling)
✅ What is Min-Max Scaling?
Min-Max Scaling, also called Normalization, transforms each feature so that its values fall within a specific range—usually 0 to 1.
It is one of the simplest and most widely used scaling techniques.
🔢 Formula
Xscaled = (X−Xmin) / (Xmax−Xmin)
This formula ensures:
Minimum value → becomes 0
Maximum value → becomes 1
Everything in between gets proportionally scaled
from sklearn.preprocessing import MinMaxScaler
minmax_scaler = MinMaxScaler()
scaled_minmax = minmax_scaler.fit_transform(numeric_data)
df_minmax = pd.DataFrame(scaled_minmax, columns=numeric_data.columns)
df_minmax
🎭 6. Encoding Categorical Variables
ML models need numbers, not text.
6.1 Label Encoding (For Target Variables)
✅ What is Label Encoding?
Label Encoding converts categories into numeric labels such as:
['HR', 'IT', 'Sales']
→ [0, 1, 2]
Each unique category gets a unique integer.
🔍 Why do we use it?
Machine learning models cannot understand text.
So LabelEncoder is used to convert text into numbers.
from sklearn.preprocessing import LabelEncoder
cat_data = pd.DataFrame({
'Department': ['HR', 'Sales', 'IT', 'HR', 'IT']
})
label_encoder = LabelEncoder()
cat_data['Dept_Label'] = label_encoder.fit_transform(cat_data['Department'])
cat_data
6.2 One-Hot Encoding (For Features)
✅ What is One-Hot Encoding?
One-Hot Encoding converts categories into separate binary columns, where:
1 = category present
0 = category not present
Example:
['HR', 'IT', 'Sales']
Becomes:
| Dept_HR | Dept_IT | Dept_Sales |
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 0 | 0 | 1 |
cat_data_ohe = pd.get_dummies(cat_data['Department'], prefix='Dept')
cat_data_ohe
🧭 7. Normalization (Unit Vector Scaling)
Normalization is a technique where we scale each data point so that the entire row (sample) has a total length of 1.
This is also called Unit Vector Scaling or Vector Normalization.
✅ Why do we use it?
Some machine-learning algorithms (like KNN, cosine similarity, text embeddings, clustering) care more about the direction of data rather than the magnitude.
Normalization ensures that:
The scale of features doesn’t affect the result
Every sample becomes equally important
We compare data based on shape/pattern, not size
📦 Example
Suppose a row of data is:
[3, 4]
Magnitude =
32+42=25=5\sqrt{3^2 + 4^2} = \sqrt{25} = 532+42=25=5
Normalized data:
[35,45]=[0.6,0.8]\left[\frac{3}{5}, \frac{4}{5}\right] = [0.6, 0.8][53,54]=[0.6,0.8]
Now the whole row has length 1.
🌟 When to use Normalization?
Use it when:
Features represent direction (e.g., text vectors, embeddings)
You are using algorithms like
KNN
K-means
Cosine similarity searches
Gradient-based neural networks
🧠 Key Idea (in one line)
Normalization doesn’t change the shape of the data — it just shrinks it so every sample has the same length.
Example:
from sklearn.preprocessing import Normalizer
import numpy as np
X = np.array([[3.0, 4.0], [1.0, 0.0], [0.0, 8.0]])
normalizer = Normalizer()
normalized_data = normalizer.fit_transform(X)
print(normalized_data)
How It Works?
For row [3, 4]:
norm = √(3² + 4²) = 5
normalized = [3/5, 4/5] = [0.6, 0.8]
Row length becomes 1 → pure direction.
🎉 Final Summary
In this blog, we covered the complete pipeline of Data Processing, including:
✔ Text, TSV, JSON file handling
✔ Scaling techniques
✔ Label & One-Hot Encoding
✔ Normalization
✔ Practical Python Examples
Data processing is the backbone of any ML or analytics task. Agar yeh part strong ho gaya, toh aapka model performance automatically improve ho jayega. 💪