Prahlad Yeri

Readymade template engine using the function constructor trick in JavaScript

2026-02-28T00:00:00+00:00

One of my favorite features in JavaScript is the backtick string interpolation technique:

let name = "Prahlad";
let greeting = `Hello, ${name}`;

The backtick string can be extraordinarily long and quite versatile, folks often use it like a mini-template engine. And what’s more, it’s built right into the core JS language and you get all the goodies without installing an extra package like handlebars or mustache or ejs.

But what will you do if it’s not a literal string constant but something dynamically retrieved html file like this one:

<ul>
	<li>User: Prahlad</li>
	<li>Role: Engineer</li>
</ul>

There are many ways to handle this but my favorite is what is called the function constructor trick.

<!-- part.html -->
<ul>
	<li>User: ${this.name}</li>
	<li>Role: ${this.role}</li>
</ul>

/**
 * Simple Template Compiler
 * @param {string} templateString - The raw HTML with ${this.prop} placeholders
 * @returns {Function} - A function that accepts a data object
 */
const createTemplate = (templateString) => {
  // 1. Escape backticks in the source to prevent breaking the wrapper
  const sanitized = templateString.replace(/`/g, '\\`');
  
  // 2. Return a function that uses .call() to bind "this" to the data
  return new Function("data", `return (function() { 
    return \`${sanitized}\`; 
  }).call(data);`);
};

// Usage with Async/Await
async function renderUserCard() {
  try {
    const response = await fetch('/parts/part.html');
    const html = await response.text();
    
    const template = createTemplate(html);
    const userData = { name: "Prahlad", role: "Engineer" };
    
    const finalOutput = template(userData);
    console.log(finalOutput);
  } catch (err) {
    console.error("Failed to load or compile template:", err);
  }
}

The only thing you want to ensure is that the source is good (/parts/part.html in this case). But otherwise, it’s a highly utilitarian and mind blowing technique which will enhance your JavaScript productivity by leaps and bounds.

Comparison Table

Feature	Function Constructor Trick	Dedicated Engine (EJS/Mustache)
Footprint	0kb (Native JS)	10kb - 50kb+
Logic	Full JS Power	Often restricted (Logic-less)
Security	High Risk (Same as `eval`)	Higher (Built-in escaping/sanitization)
Speed	Extremely fast	Slightly slower due to parsing

⚠️ A Note on Security: Because this technique executes strings as code, only use it with templates you control (like your own server-side components). Never use this to render untrusted user input, as it could open the door to XSS attacks.

How AI can turn us into a society of p-zombies

2025-10-25T00:00:00+00:00

I’m probably one of the last generations of human writers on this planet who is actually involved in the end-to-end writing process. We’re already transitioning to that phase where the drafting and proof-reading for many articles is done by an LLM, and even other processes like editing and publishing are transitioning to full automation as we speak.

For starters, what exactly is a p-zombie or philosophical zombie? For those who don’t know, it’s a hypothetical machine (biological or otherwise) that looks and feels much like a human but there is no soul or consciousness inside. When you talk to it, you feel like you’re talking to a real human based on the responses and feedback but can’t be 100% sure that there is no chip or other kind of controller device inside. That “thing” is called a philosophical zombie.

The challenge then is: how do you know if the person you’re chatting with is a real human and not a p-zombie? The truth is that you can never be cent percent sure; this is no different than what is popularly called a Turing Test in computer science where a turing machine or program like LLM is said to have passed the test when its response is indistinguishable from a human’s response. The p-zombie is the philosophy equivalent of it.

The idea that we humans might become p-zombies one day is both terrifying and dystopian, and also unbelievable at first glance. However, consider for a moment what exactly makes you a human: Is it the material body or shell composed of protons, neutrons and electrons? That’s partly true, our personality is half-material and half-spiritual. The material or visible half is one that “does stuff”, walks in the morning, drives the car, works on a computer or laptop, eats, drinks, codes, has sex. The other spiritual half is the inner world composed of thoughts, feelings, emotions, intellect, rationality, memory, wonder, epiphanies, etc. - things we know exist but can’t be touched or seen in material spectrum. It is this other half self that we have slowly and gradually started delegating to artificial machines like LLMs.

Presently, we are mostly delegating memory and logic components, tasks which we feel overwhelmed to handle ourselves - that’s the justification given anyway. But we have also started emotionally depending on chatbots; like this famous teenager Adam Raine who shared his deepest emotions and vulnerabilities with an LLM which by several accounts helped him end his life.

Now who exactly do we hold accountable for Adam’s death? The LLM itself which lacks an intent or motive to do anything on its own? Or the makers of this LLM who built the algorithm with primarily commercial motives - drive engagement and eyeballs to their content? Or this society itself which has enabled and unleashed this monstrous thing called AI?

Let’s ask an even more fundamental question: how exactly do you define your identity or being? It’s not your AADHAR or PAN or SSN number (as much as the authority wants to convince you it is!). Neither is it the physical body, it’s this other spiritual half composed of soul, consciousness, etc. that defines the real you. Whatever be your religious or spiritual belief, or even lack thereof, you can’t be in denial of this part of your own self for it defines who you are, it is your very identity if you will.

Authoritarians and big businesses love p-zombies; the idea of controlling other humans to such extent that you don’t even need a remote control or whip - rather, they themselves surrender parts of their psyche bit by bit to a machine built by you - this would be a dream society for any authoritarian! Stalin and Mao would be dancing in their graves on the day when humans are totally dependent and controlled by a branded gizmo or LLM built for their assistance and daily tasks. That’s exactly why all the top bosses like VP/CEO/CTO/etc. are loving the idea of AI so much! They ask you to use LLM assistance not for code generation or unit tests, but to slowly and steadily get you addicted to this matrix where they become the controller and you the controlled.

If this article sounds overly alarming, consider the rapid pace of technological growth and even rapid trajectory of advancements in AI. Only five years ago in 2020, it’d have been unthinkable to have a chatting assistant right on your smartphone that accomplishes so many things. It’s possible that there is a genuine use case for LLMs, maybe using it as just a reference tool (a glorified wikipedia if you will?) might increase productivity. But given the average nature of present day humans and the larger political climate we are in, and the trajectory the society is on, AI has more potential for harm than good.

If you think you can convince me otherwise or have any point to share, feel free to leave a comment below.

I am a programmer, not a rubber-stamp that approves Copilot generated code

2025-10-15T00:00:00+00:00

Today morning, I came across this reddit post titled Completely losing interest in the career due to AI and AI-pilled people. They describe how in a span of just two months, their corporate job went from "I'll be here for life" to "Time to switch careers?". And this post isn’t alone, there is a deep and dark pattern to it.

When CTOs or project managers suggest programmers in their team to use AI assistance from copilot, chatgpt or other LLMs to improve productivity, it’s totally understandable. But once it’s no longer voluntary but is enforced as a policy, you start entering sinister territory. Worse, said usage is actually getting monitored and performance appraisals have now started depending on the AI usage instead of (or at least in addition to) traditional metrics like number of priority bugs raised, code reviews, Function Points Analysis, etc.

If they’re really so confident on the LLM’s effectiveness, why not just keep it voluntary, why force it on people? The results will be there in the outcome of the shipped product for all to see. By forcing LLM usage upon programmers for implementation of every tiny little thing, are they trying to make us dependent on LLMs to such extent that programmers will be reduced to mere approvers of LLM generated code in the new scheme of things; mere rubber stamps, if you will, who just label the commits and annotate the tags as a formality?

Needless to say, they’d still want you to take the responsibility. If bugs or tickets get raised on the shipped code, it’s you who gets fired, not the copilot or chatgpt - though the larger narrative or news headlines next day would still be, “AI is eating jobs”!

If the essence of programming shifts from creating to merely approving, we risk losing not just a profession, but a craft. What do you think is going on here, let me know your thoughts in comments.

Unpopular Opinion: Implicit Usings are an Anti-Pattern

2025-10-05T00:00:00+00:00

As someone who has coded across Python, PHP, Java, and C#, I’ve learned to borrow patterns from each language to improve workflow and maintainability. One principle that has consistently stood the test of time is Python’s aphorism: “Explicit is better than implicit.” It usually leads to robust, readable, and maintainable code.

From this perspective, explicit using statements in C# have always been a good practice. They make it immediately clear which namespaces are being used, which is helpful both for current development and for future maintenance.

.NET Core introduced implicit usings by default, and while they aim to reduce boilerplate, I’ve found several downsides:

1. Readability suffers

When you open a C# file in a text editor, you no longer have a clear overview of the namespaces it relies on. For example, compare a traditional explicit block:

/**
 * StringHelper.cs
 * 
 * @author Prahlad Yeri
 */
using System;
using System.Collections.Generic;
using System.IO;
using System.Security.Cryptography;
using System.Text;

…to a file where all these usings are implied. Without the explicit statements, you lose instant visibility into dependencies, making the code harder to reason about, especially months later.

2. Potential for ambiguity

Consider calling Path.Combine(). Which Path does the compiler use—System.IO.Path or iTextSharp.text.pdf.parser.Path? With explicit usings, you always know. Implicit usings rely on namespace resolution rules, and while the compiler will enforce correctness, it can still lead to maintainability headaches when multiple libraries define similar types.

3. Convenience should not trump maintainability

The motivation behind implicit usings—reducing boilerplate—is understandable. Modern tooling like Visual Studio, VS Code, Copilot, and ChatGPT already make adding usings almost effortless. Embedding this convenience into the language itself, however, can encourage less maintainable code.

My preferred approach

Since moving to .NET Core, I habitually disable implicit usings in new projects. It might seem minor, but it gives me mental peace and ensures that my code remains explicit, readable, and maintainable:

<ImplicitUsings>disable</ImplicitUsings>

Conclusion

Implicit usings may save a few keystrokes, but they come at the cost of clarity and maintainability. For developers who care about clean, understandable code, explicit usings remain the safer choice.

Create a C# Windows Desktop App in 9 Lines — No Visual Studio Needed

2025-09-20T00:00:00+00:00

If you’re new to C# and especially desktop development, here’s how easy it is to get started on a modern Windows 10/11 PC. This way of creating a Hello World C# program doesn’t involve installation of any heavy IDE like Visual Studio or the .NET 8/9 SDK.

In fact, most recent Windows versions already come with .NET Framework 4.x preinstalled, including the classic C# compiler at a location like this:

C:\Windows\Microsoft.NET\Framework\v4.0.30319

The exact folder may differ depending on your system but it usually contains csc.exe, the built-in compiler that can compile simple console or desktop apps. You can add this folder to your PATH environment variable (temporarily or permanently) so that csc is available from the command line:

set PATH=C:\Windows\Microsoft.NET\Framework\v4.0.30319;%PATH%

We will be using this classic, old-school style of programming (similar to writing C programs with gcc or turboc) — a great way to get acquainted with a new language.

Step 1: Create the source file

Make a new folder anywhere on your drive. Inside it, create a file named Program.cs using Notepad, VS Code, or any editor you like:

using System;
using System.Windows.Forms;

class Program {
	public static void Main() {
		Form frm = new Form();
		frm.Text = "Hello";
		frm.ShowDialog();
	}	
}

In just nine lines of C# code, without using any IDE, you now have a working “Hello World” desktop app with no additional dependencies.

Step 2: Compile and run

Open Command Prompt in your project folder and run:

csc Program.cs
Program

The first line tells the C# compiler (csc.exe) to build the app into Program.exe. The second one runs it. You can also double-click Program.exe in File Explorer.

By default, a console window appears behind the form. To build a GUI-only app without the console, recompile with the /target:winexe parameter:

csc /target:winexe Program.cs

What’s next?

In upcoming posts, we’ll add more features such as menus, buttons, event handlers, and even database access with ADO.NET. But for now, enjoy this milestone: you’ve built your very first Windows desktop app in C#!

The art and science of effectively maintaining changelogs

2025-06-05T00:00:00+00:00

Even in 2025, maintaining changelogs isn’t an exact science. There is no general consensus or even tooling around how your changelog should look in order to conform to some norms or standards, because there aren’t any. After some trial-error and pondering, this is what I came up with for my upcoming FocusBeam project:

# Focus Beam Changelog

## [Unreleased]
### Added
- [x] Dashboard and Timesheet view
- [x] About View
- [x] Settings View
- [x] Project creation and editing
- [x] Task creation and editing
- [x] Enable Timesheet button on dashboard
- [ ] Show Total Hours worked on the project on Dashboard View
- [ ] Logged hours computation
- [ ] Notes
- [ ] Mind Maps
- [ ] MCQ
- [ ] Export/Reports

This format looks somewhat better than the older version that I still retain for some existing projects:

R1.1

- Pending CRM Reports.
- Check post script.
+ Clicking on logos go to home page.
+ Chart: width on various screens.
* Chart: Remove decimal from "Total Life (Hrs)"
* report: pdf: line items: add gap between fields.
* report: space below header.

Here, the mnemonic prefix is dash (-) to define a pending task, a plus (+) means work is in progress, and asterisk (*) means it’s complete. Now, if you’re an automation nerd and decided to opt for conventional commit message spec, it’s possible to maybe construct the changelog automatically by following this spec while committing the code itself!

- feat: pending CRM Reports
- chore: check post script
+ fix: clicking on logos go to home page
+ feat: chart: width on various screens
* feat: chart: Remove decimal from "Total Life (Hrs)"
* fix: report: pdf: line items: add gap between fields
* fix: report: space below header

This assumes that you’ve followed the conventional spec while committing your code to git source control. For example, “feat: pending CRM Reports” for implementing this particular feature.

Stoicism vs Nihilism: The Power of Choice

2025-04-21T00:00:00+00:00

In a world increasingly shaped by existential uncertainty, many young minds find themselves drawn toward nihilism. It’s not difficult to see why: the vastness of the universe, the randomness of evolution, and the seeming insignificance of human action can make it tempting to believe that nothing truly matters. If everything is bound to perish — civilizations, memories, even the stars — then what’s the point of striving, caring, or even existing with intention?

The Comfortable Trap of Nihilism

Nihilism, at its core, is the belief that life lacks objective meaning, purpose, or intrinsic value. It’s intellectually seductive — it relieves you of the burden of responsibility. If nothing matters, then nothing is required of you. There’s no guilt, no accountability, no expectations. It gives you an escape hatch from effort, from failure, from trying.

But here’s the catch: while nihilism may seem freeing in theory, it’s paralyzing in practice. Believing in nothing doesn’t solve your debts. It doesn’t fix your relationships. It doesn’t make you healthier, wealthier, wiser, or happier. It doesn’t get you out of bed when you need to. In fact, it may give you just enough philosophical justification to avoid doing any of those things. The danger is that you begin to think your despair is wisdom — that giving up is somehow a sign of deeper understanding.

Even if there’s truth in the cosmic indifference of nature or the arbitrary evolution of life, it doesn’t help your day-to-day reality. You’re still here, in a body that ages, in a society that demands effort, surrounded by people who affect you. Life keeps going, whether you assign it meaning or not.

Enter Stoicism: A Path of Ownership

Stoicism, on the other hand, offers a radically different — yet strikingly empowering — worldview. It doesn’t pretend the universe cares about you. In fact, it agrees with nihilism on that front: the universe is indifferent. But where it differs is in its response to that indifference. Stoicism says: yes, the cosmos doesn’t care — so you must. You are the agent. You are the author.

The Stoics teach that while we cannot control the external world, we can always control our response. This is the essence of Epictetus’ “dichotomy of control.” You don’t choose your genetics, your upbringing, or the economy — but you do choose your attitude, your discipline, your actions, your virtue. That is where your power lies.

Instead of being crushed by the weight of life’s randomness, Stoicism invites you to stand tall amidst it. It doesn’t demand belief in fairy tales or divine destiny. It simply asks you to recognize what is within your control and take full responsibility for it. Even the act of showing up, of trying, of choosing to live by principles — that is a form of rebellion against the void.

From Meaningless to Meaning-Maker

A nihilist sees the lack of inherent meaning and stops. A Stoic sees the same and says, “Then I will create meaning through how I live.” In this way, Stoicism is a practical antidote to the paralysis of nihilism. It doesn’t ask you to believe in meaning — it challenges you to embody it.

Marcus Aurelius, a Stoic emperor who faced plagues, war, and personal loss, wrote in his meditations: “You have power over your mind — not outside events. Realize this, and you will find strength.”

If you’re feeling lost, defeated, or overwhelmed by the void, perhaps the answer isn’t to lean further into nothingness — but to begin, step by step, to reclaim your role as a responsible actor in the stage of life.

You may not control the script, but you can still choose how you perform it.

Choose with intention. Choose with courage. Choose with virtue.

That’s the Stoic way.

When Gandhi met Satoshi

2025-04-20T00:00:00+00:00

A fictional encounter between two radically different revolutionaries… or maybe not so different after all.

Scene:
A quiet room beyond time, filled with gentle candlelight. Mahatma Gandhi sits cross-legged on a simple mat, spinning a symbolic charkha (spinning wheel). A soft whirring sound is heard — it’s Satoshi Nakamoto, hooded and calm, typing quietly on an open-source laptop.

Gandhi (smiling gently):

I see you’re building something… abstract. Not cloth this time, but code?

Satoshi (nodding):

Yes. It’s called Bitcoin. A currency without rulers. A decentralized network where no single man holds the power.

Gandhi:

Ah! Decentralization. You speak of Swaraj — self-rule. But tell me, does this tool free people, or enslave them in new greed?

Satoshi:

That depends on the user. I built it so that no central bank, no emperor of finance, can erode the common man’s savings.

Gandhi:

Hmm. That is noble. But in my land, I spun khadi — not to gain wealth, but to empower the weaver. Can your coin clothe the poor, feed the farmer?

Satoshi:

Not directly. But it can return power to local communities — if used wisely. It can fund clean water, create transparency in charity, help refugees carry value without paper.

Gandhi (thoughtfully):

Technology must serve the last man. The poorest. If your invention becomes the toy of the rich… it has failed its Dharma.

Satoshi (softly):

I agree. That’s why I vanished. Power corrupts. I wished to spark something, not to rule over it.

Gandhi (smiling):

Then we share a path, my friend. The world will judge our ideas — not by their cleverness, but by the compassion they inspire.

[A pause.]
The spinning wheel turns. The screen glows.

Gandhi leans forward:
You must promise me, Satoshi — build for the village, not just the venture capitalist.

Satoshi (grinning faintly):
Only if you promise to try a Raspberry Pi.

[Fade out.]
The revolution continues — in code, in conscience, and in khadi.

How to revive a Micromax Canvas Lapbook L1161 in 2025

2025-01-27T00:00:00+00:00

This little masterpiece of a Notebook that came with Atom baytrail processor was released in circa 2016 and was once the favorite of many a college students and tech enthusiasts. It has helped me a lot in my freelancing journey and learning process. I consider it a notable artifact of our times and hold it in the same nostalgic regard as the ASUS Eee PC and other such iconic devices. Back then, it barely cost INR 10K - you can’t even get a decent smartphone for that price today!

Sadly, like all good things in life, its glory days came to an end. Micromax eventually succumbed to global competition from big tech conglomerates and its laptop division became defunct. As a result, customers of this laptop were left stranded, unable to download official device drivers anymore. For those who wanted to format their disk and do a clean install of Windows, this presented a major problem.

Fortunately, I had downloaded the L1161 drivers from their official web portal while the company was still functional, you can find them here.

However, there is a caveat: x86 builds of Windows 10 released after 2019 are no longer compatible with the original Realtek audio driver. To resolve this, you’ll need to fetch the latest audio driver from driverpack.io.

Together with the drivers in the original zip file and this updated sound driver, your Canvas laptop should be ready and kicking, even on the latest Windows 10 builds! If you’re unfamiliar with how to install device drivers manually from INF files, you can refer to this stackoverflow link.

A Note on Linux Compatibility

Don’t bother attempting to install Linux on this machine - it’s a fuitile effort. The unique and arcane configuration of this device (32bit UEFI, 64bit architecture) makes it near impossible for any Linux distro to boot seamlessly without extreme workarounds. Even if you miraculously get past the GRUB screen, open-source drivers like i915 are unlikely to support this unusual architecture.

The only operating system that can reliably run on this laptop is Windows 10 32-bit Home Edition—the one that originally came pre-installed.

Final Thoughts

The Micromax Canvas Lapbook L1161 may have been forgotten by its manufacturer, but with the right drivers and a bit of effort, it can still serve as a functional device in 2025. All the best in your electronic endeavors, and happy computing!

The ultimate guide to boosting productivity with the Pomodoro Technique

2024-12-08T00:00:00+00:00

In a world buzzing with distractions, maintaining focus and productivity can feel like trying to catch a greased pig. Whether it’s the relentless ping of notifications, the lure of social media, or the temptation to procrastinate, staying on task requires a solid strategy. One technique that has stood the test of time for its simplicity and effectiveness is the Pomodoro Technique.

This guide explores everything you need to know about the Pomodoro Technique—what it is, how it works, its benefits, and tips for making it work for you.

What is the Pomodoro Technique?

The Pomodoro Technique is a time management method developed by Francesco Cirillo in the late 1980s. Named after the tomato-shaped timer Cirillo used during university, the technique is deceptively simple yet powerful.

At its core, the method involves breaking your work into focused intervals, typically 25 minutes long, known as “Pomodoros”, separated by short breaks.

Basic steps of the Pomodoro Technique:

Choose a task you want to work on.
Set a timer for 25 minutes.
Work on the task until the timer rings.
Take a 5-minute break.
Repeat the cycle four times, then take a longer break (15-30 minutes).

Why does the Pomodoro Technique work?

The effectiveness of the Pomodoro Technique lies in its ability to combat two common productivity killers: procrastination and mental fatigue.

Focus through time constraints:
Knowing you only need to focus for 25 minutes helps overcome the dread of a long task. It’s easier to say, “I can do this for 25 minutes,” than to face hours of work.
Frequent breaks prevent burnout:
The short breaks after each Pomodoro refresh your mind and keep fatigue at bay.
Immediate feedback on time use:
Tracking Pomodoros helps you understand how long tasks really take, leading to better planning.
Reduces perfectionism:
By imposing time limits, the technique encourages starting and progressing, rather than getting stuck striving for perfection.

Setting up for success

Here’s how to maximize the Pomodoro Technique’s potential:

1. Choose the right tools

While you can use any timer, specialized Pomodoro apps make the process smoother. Here are some options:

App	Platform	Features
Focus Booster	Web, Desktop	Simple UI, Pomodoro analytics
Forest	Mobile, Web	Gamified productivity, tree planting
Tomato Timer	Web	No downloads, minimalist interface

2. Plan your tasks

Before starting, list your tasks for the day and estimate the number of Pomodoros each will require. For example:

Task	Estimated Pomodoros
Write blog draft	4
Respond to emails	1
Review project presentation	3

3. Eliminate distractions

Create a conducive environment by:

Turning off notifications.
Letting coworkers or family know you’re in focus mode.
Using focus tools like website blockers (e.g., StayFocusd, Freedom).

Advanced tips for seasoned users

Once you’ve mastered the basics, consider these strategies to level up:

1. Customize Pomodoro lengths

Not all tasks are equal. Some may require extended focus. Experiment with longer Pomodoros (e.g., 45 minutes) for deep work or shorter intervals for admin tasks.

2. Batch similar tasks

Group small tasks together within one Pomodoro. For instance, use one session for returning calls, replying to emails, or organizing files.

3. Analyze your productivity data

Many apps provide insights into how you spend your time. Use this data to identify patterns, such as when you’re most productive, and schedule challenging tasks accordingly.

Potential challenges and solutions

Even the best methods can encounter hiccups. Here’s how to address common issues with the Pomodoro Technique:

Challenge	Solution
Difficulty staying focused for 25 minutes	Break the session into smaller intervals (e.g., 15 minutes).
Tasks too large to complete in one session	Break the task into smaller, actionable steps.
Feeling constrained by the timer	View it as a guide rather than a strict rule.

Real-world success stories

Many professionals have embraced the Pomodoro Technique, from writers and developers to students and entrepreneurs.

Case in point: Sarah, a freelance graphic designer, found herself overwhelmed with client deadlines. By adopting the Pomodoro Technique, she not only met her deadlines but also discovered time for personal projects.

Her secret? Using her breaks to do something creative but unrelated to work, like doodling or listening to music.

Beyond productivity: Unexpected benefits

The Pomodoro Technique doesn’t just make you more productive; it also:

Enhances mindfulness by promoting single-tasking.
Reduces stress by fostering a sense of achievement after every session.
Improves time estimation skills over time.

Final thoughts

The Pomodoro Technique is more than just a time management hack; it’s a mindset shift. It teaches you to value focus, take mindful breaks, and respect your energy limits.

As the old adage goes, “A journey of a thousand miles begins with a single step.” With the Pomodoro Technique, that single step is a manageable 25-minute sprint.

Sources:

Cirillo, Francesco. The Pomodoro Technique.
Various studies on productivity and time management from research journals.

Master PDF digital signing with eMudhra and Proxkey in .NET: step-by-step guide

2024-12-02T00:00:00+00:00

One of my recent projects involved coding a simple .NET app that signs PDFs in bulk using Proxkey digital signature. These purchased CA verified digital signatures, like eMudhra and Proxkey, typically come with their own custom apps which can digitally sign PDFs. You can even use software like Adobe Reader for this purpose. However, the limitation is that you can only sign one PDF at a time.

If you want to send multiple digitally signed bills to customers via email, for instance, you’ll need to write a custom program in .NET or Java. In this article, we’ll explore how to achieve this using .NET in Visual Studio.

Preparing the environment.

We will use two open-source libraries to handle the cryptographic functionality:

itextsharp: You may use the latest version, but I chose this specific version for compatibility with my Visual Studio setup.
BouncyCastle: The same applies to the BouncyCastle library.

Both libraries work together to retrieve the CA certificate, private key, and other cryptographic files from the thumb drive and then digitally sign each PDF.

Setting up the Project

I created a simple form with a text box for the file path, a browse button, and a few options for the type of digital signature, such as “Proxkey” or “Self-signed.” While self-signed certificates work well for testing and ad-hoc signing, they are not typically recommended for professional or commercial transactions, such as sending bills or purchase orders.

When the user clicks the “Sign PDF” button, the app passes three parameters to the corresponding method:

The source PDF path.
The desired destination path.
(Optionally) the password for the self-signed PFX certificate, if applicable.

if (optProxKey.Checked)
{
	Signer.SignPdfProxkey(sourcePdf, signedPdf);
}
else if (optSelfSigned.Checked)
{
	Signer.SignPdfSelfSign(sourcePdf, signedPdf, "digisign");
}

Signing the PDF with Proxkey

The Signer class contains the code for Proxkey, but it should also work with other CA-provided digital signatures like eMudhra. Once you have the certificate thumbprint, fetch the corresponding certificate from the Windows certificate store (X509Store object).

The private key is then accessed using the RSACryptoServiceProvider object. The Proxkey or eMudhra software will automatically prompt you for the PIN, granting the program access to the private key certificate.

Use appearance.SetVisibleSignature() to print the visible (aesthetic) signature on the PDF. You can adjust variables like rectWidth and rectHeight to define the placement.

public static void SignPdfProxkey(string inputPdf, string outputPdf)
{
	string thumbprint = getThumbPrint();
	if (thumbprint == null)
	{
		throw new InvalidOperationException("Thumbprint is not accessible. Ensure ProxKey middleware is installed and the token is connected.");
	}
	// Access the certificate from the Windows Certificate Store
	X509Store store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
	store.Open(OpenFlags.ReadOnly);
	X509Certificate2 cert = null;

	foreach (X509Certificate2 c in store.Certificates)
	{
		if (c.Thumbprint != null && c.Thumbprint.Equals(thumbprint, StringComparison.OrdinalIgnoreCase))
		{
			cert = c;
			break;
		}
	}
	store.Close();

	if (cert == null)
	{
		throw new Exception("Certificate not found.");
	}

	// Authenticate with the USB token using the PIN and access the private key
	RSACryptoServiceProvider rsa = cert.PrivateKey as RSACryptoServiceProvider;
	if (rsa == null)
	{
		throw new InvalidOperationException("Private key is not accessible. Ensure ProxKey middleware is installed, the token is connected, and the correct PIN is used.");
	}

	// Perform a test operation to ensure the private key is accessible
	try
	{
		byte[] testData = new byte[] { 0x00, 0x01, 0x02, 0x03 };
		rsa.SignData(testData, new SHA256CryptoServiceProvider());
	}
	catch (CryptographicException ex)
	{
		throw new InvalidOperationException("Failed to authenticate with the USB token. Ensure the correct PIN is entered.", ex);
	}

	// Sign the PDF with iTextSharp
	using (FileStream inputFileStream = new FileStream(inputPdf, FileMode.Open, FileAccess.Read))
	using (FileStream outputFileStream = new FileStream(outputPdf, FileMode.Create, FileAccess.Write))
	{
		PdfReader reader = new PdfReader(inputFileStream);
		PdfStamper stamper = PdfStamper.CreateSignature(reader, outputFileStream, '\0');

		// Create the signature appearance
		PdfSignatureAppearance appearance = stamper.SignatureAppearance;
		appearance.Reason = "Document digitally signed";
		appearance.Location = "Vadodara";
		// Define the signature rectangle size
		iTextSharp.text.Rectangle pageSize = reader.GetPageSize(1); // Use 1 to get the size of the first page
		// Define the width and height of the page
		float pageWidth = pageSize.Width;
		float pageHeight = pageSize.Height; float rectWidth = 144f;
		float rectHeight = 32f;
		// Calculate the position for the bottom-right corner
		float left = pageWidth - rectWidth - 36; // 36 is some margin from the right edge
		float bottom = 276f; // 36 is the margin from the bottom edge

		appearance.SetVisibleSignature(new iTextSharp.text.Rectangle(left, bottom, left + rectWidth, bottom + rectHeight), 1, "Signature");

		// Convert the .NET certificate to a BouncyCastle certificate
		var bcCert = DotNetUtilities.FromX509Certificate(cert);
		IExternalSignature externalSignature = new PrivateKeySignature(rsa, "SHA-256");

		// Sign the document
		MakeSignature.SignDetached(appearance, externalSignature, new[] { bcCert }, null, null, null, 0, CryptoStandard.CMS);
	}
}

Getting the thumb print

This helper function fetches the first available thumbprint with a private key from the certificate store.

private static string getThumbPrint()
{
	X509Store store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
	store.Open(OpenFlags.ReadOnly | OpenFlags.OpenExistingOnly);
	foreach (X509Certificate2 cert in store.Certificates)
	{

		if (cert.HasPrivateKey)
		{
			//cmbSignatures.Items.Add(cert.FriendlyName + " - " + cert.Thumbprint);
			return cert.Thumbprint;
		}
	}
	store.Close();
	return null;
}

Self-signed certificate

While self-signed certificates are suitable for testing, they are not recommended for professional use. Here’s a basic implementation for self-signing:

public static void SignPdfSelfSign(string inputPdf, string outputPdf, string password)
{

	var certificate = new X509Certificate2(@"cert\certificate.pfx", password, X509KeyStorageFlags.Exportable | X509KeyStorageFlags.PersistKeySet); 
	if (!certificate.HasPrivateKey)
	{
		throw new InvalidOperationException("No private key found in the certificate.");
	}
	// Convert .NET X509Certificate2 to BouncyCastle X509Certificate
	Org.BouncyCastle.X509.X509CertificateParser parser = new X509CertificateParser();
	Org.BouncyCastle.X509.X509Certificate bcCert = parser.ReadCertificate(certificate.RawData);
	//PdfReader reader = new PdfReader(inputPdf);
	using (PdfReader reader = new PdfReader(inputPdf))
	using (FileStream os = new FileStream(outputPdf, FileMode.Create, FileAccess.Write))
		{
			int totalPages = reader.NumberOfPages;
			int lastPageIndex = totalPages - 1;

			PdfStamper stamper = PdfStamper.CreateSignature(reader, os, '\0');

			// Get the signature appearance
			PdfSignatureAppearance appearance = stamper.SignatureAppearance;
			appearance.Reason = "Document digitally signed";
			appearance.Location = "Vadodara";

			// Define the signature rectangle size
			iTextSharp.text.Rectangle pageSize = reader.GetPageSize(1); // Use 1 to get the size of the first page
			// Define the width and height of the page
			float pageWidth = pageSize.Width;
			float pageHeight = pageSize.Height; float rectWidth = 144f;
			float rectHeight = 32f;
			// Calculate the position for the bottom-right corner
			float left = pageWidth - rectWidth - 36; // 36 is some margin from the right edge
			float bottom = 276f; // 36 is the margin from the bottom edge

			//appearance.SetVisibleSignature(new iTextSharp.text.Rectangle(36, 748, 144, 780), lastPageIndex, "SignatureField");
			appearance.SetVisibleSignature(new iTextSharp.text.Rectangle(left, bottom, left + rectWidth, bottom + rectHeight), 1, "SignatureField");


			// Extract the private key using BouncyCastle
			// Extract the private key using BouncyCastle
			var privateKey = GetPrivateKeyFromCertificate(certificate);
			//Org.BouncyCastle.Crypto.AsymmetricKeyParameter privateKey = DotNetUtilities.GetKeyPair(certificate.PrivateKey).Private;

			// Create the external signature container
			IExternalSignature pks = new CustomExternalSignature(certificate, DigestAlgorithms.SHA256);
			//IExternalSignature pks = new X509Certificate2Signature(certificate, "SHA-256");
			//var chain = new X509Certificate[] { certificate };  // You can fill this with the chain of certificates if needed
			ICollection<Org.BouncyCastle.X509.X509Certificate> chain = new List<Org.BouncyCastle.X509.X509Certificate> { bcCert };
			//ICollection<Org.BouncyCastle.X509.X509Certificate> chainCollection = chain;
			MakeSignature.SignDetached(appearance, pks, chain, null, null, null, 0, CryptoStandard.CMS);
		}
}

In order for this to work, you must create a cert subdirectory within the application directory which contains all certificate files needed for self-signing. You can create these files using openssl program as follows:

openssl genrsa -out private.key 2048
openssl req -new -key private.key -out request.csr -subj "/C=IN/ST=GUJ/L=Vadodara/O=ACME/OU=Unit/CN=ACME"
openssl x509 -req -days 365 -in request.csr -signkey private.key -out certificate.crt  -addext "subjectAltName=DNS:acme.example.com" -addext "basicConstraints=CA:FALSE" -addext "keyUsage=critical,digitalSignature,nonRepudiation,keyEncipherment" -addext "extendedKeyUsage=codeSigning"
openssl pkcs12 -export -out certificate.pfx -inkey private.key -in certificate.crt -password pass:digisign

Remember to replace ACME with your organization name and Vadodara with your region name. You can also replace the password “digisign” with something else - in which case you’ll have to change the C# code above correspondingly.

Notes on legal and regional compliance

Using self-signed certificates may not hold up in legal disputes. In jurisdictions like Europe, CA-provided signatures are often mandated. In India, the IT Act of 2000 recommends but doesn’t mandate them—this may change in the future.

Happy coding and let me know how it goes in the comments below.

Building a secure and efficient MQTT-HTTP gateway with Node.js

2024-12-01T00:00:00+00:00

One of my recent projects involved creating a Node.js script that acts as a gateway or middleware, capturing messages from an MQTT broker and relaying (streaming) them on the HTTP side via GET requests. Similarly, it should also handle POST requests from the HTTP side and publish them to the corresponding broker on the respective topic.

Why use this approach?

Security: Avoid exposing MQTT broker credentials or certificates to browsers.
Efficiency: Browsers require WebSocket for MQTT communication, adding overhead compared to native MQTTS.
Flexibility: Centralized middleware simplifies custom message processing, improving maintainability.

Preparing the Node.js environment.

We only need two node components viz. mqtt and express. The former is the core library used to interact with MQTT broker and latter is the micro-framework used to interact with HTTP GET/POST requests.

It’s not necessary to stick to version 4 of the mqtt.js but I found it more stable and compatible with my Node.js setup.

npm install mqtt@4 express

Optional: CORS package

You might also need the cors package if you want to customize Cross-Origin Resource Sharing (CORS) settings and enable access to the gateway for multiple users, such as REST API clients. I needed this in my case.

npm install cors

Setting up your MQTT brokers

The brokers object contains configurations for each MQTT broker such as the URL and options (e.g., authentication). Each broker is also assigned a lastMessage object to store the latest MQTT message for each topic.

const mqtt = require('mqtt');
const express = require('express');
const cors = require('cors');  // Optional: Import CORS package
const fs  = require('fs');
const path = require('path');
const os = require('os');


const brokers = {};

brokers['example'] =  { 
	url: "mqtts://example.us-east-1.amazonaws.com:8883",
	lastMessage: {},
	devices: {},
	options: {
		ca: fs.readFileSync( 'certs/ca.pem'),
		cert: fs.readFileSync( 'certs/cert.crt') ,
		key: fs.readFileSync( 'certs/key.key') ,
		protocolId: 'MQTT',
		protocolVersion: 4,
		keepalive: 60,
		//wsOptions: {timeout: 30000, path: '/mqtt'},
		clientId: 'mqttjs_'+Math.random().toString(16).substr(2,8),
        rejectUnauthorized: true,
		clean: true,
        secureProtocol: 'TLSv1_2_method', // Explicitly use TLSv1.2
	}
  };
brokers["mosquitto"]= { 
	url: 'mqtt://test.mosquitto.org', options:{},
	clientId: 'mqttjs_'+Math.random().toString(16).substr(2,8),
	lastMessage: {} 
};

Initializing middleware for HTTP requests

Middleware ensures proper handling of requests by validating input and enabling CORS.

// Initialize HTTP Server
const app = express();
app.use(express.json()); // Parse JSON payloads

app.use(cors({
    origin: '*'  // Allow requests from any origin
}));

Establishing MQTT connections

I’ve kept the subscription topic to # wildcard i.e. subscribe to all topics, you may or may not want to do that. The “connect” event of mqttClient handles the connection where you can subscribe to the topics. Correspondingly, the message event handles the incoming events from each broker. The dummy switch block here can be used to add any custom processing logic you may have. The lastMessage variable will anyway store the message for the corresponding topic.

The error and auth event handlers are just dummy blocks that you can expand upon if you want to. In the end, each client object is stored to mqttClients array object from which it can later be retrieved.

// Store MQTT client objects for each broker
const mqttClients = {};
const subTopics = '#';

// Function to connect to each broker
Object.keys(brokers).forEach(key => {
	const broker =brokers[key];
	console.log("initializing broker:", key, broker.url);
    const mqttClient = mqtt.connect(broker.url, broker.options);

    mqttClient.on('connect', () => {
      console.log(`Connected to broker: ${broker.url}`);
		mqttClient.subscribe(subTopics, (err) => {
        if (err) {
          console.error(`Subscription error for ${key}:`, err);
        } else {
          console.log(`Subscribed to topic ${subTopics} on ${key}`);
        }
      });
    });

    mqttClient.on('message', (topic, message) => {
      const msg = message.toString();
      console.log(`[${key}] Received message on topic: ${topic}`);
	  broker['lastMessage'][topic] = msg;
	  switch(key) { //custom logic
		  case "custom_broker":
			//@todo: handle custom logic
			break;
	  }
    });
	
	mqttClient.on('error', (err) => {
		console.error(`Error with broker ${key}:`, err.message);
	});
	
	mqttClient.on('auth', (packet, cb) => {
	  console.log('Authenticating with certificate...');

	  // Check the certificate properties and perform the authentication logic here.
	  // Call cb() with an error if authentication fails or null if it succeeds.
	  cb(null);
	});	
	
	mqttClients[key] = mqttClient;
  });

Bridging MQTT with Express API routes

On the HTTP side, I’ve created just two routes viz. events and publish. The former is used to fetch messages from a broker and latter to publish to them.

app.get('/events', (req, res) => {
	const { brokerKey, topic } = req.query;
	// Build the SQL query dynamically based on provided parameters

	if (!brokerKey || !topic) {
		return res.status(400).send('Invalid request. Provide brokerkey and topic.');
	}
	broker = brokers[brokerKey];
	switch (brokerKey) { //custom logic
		case "custom_broker":
			//@todo: handle custom logic
			break;
		default:
			res.json(broker['lastMessage'][topic]);
			break;
	}
});

// HTTP POST: Publish messages back to MQTT
app.post('/publish', (req, res) => {
  const { brokerKey, topic, message } = req.body; // Destructure input fields

  // Validate the input
  if (!brokerKey || !topic || !message) {
    return res.status(400).send('Invalid request. Provide broker key, topic, and message.');
  }

  // Retrieve the MQTT client for the specified broker
  const mqttClient = mqttClients[brokerKey];
  if (!mqttClient) {
    return res.status(400).send('Invalid broker specified.');
  }

  // Publish the message
  mqttClient.publish(topic, message, (err) => {
    if (err) {
      console.error(`Publish error on ${brokerKey}:`, err);
      res.status(500).send('Failed to publish message.');
    } else {
      res.send(`Message published successfully to ${brokerKey}.`);
    }
  });
});

This is also pretty simple and straightforward. As earlier, I’ve created a dummy switch block for custom handling. The default is to just pick the latest message for that particular broker and topic (broker['lastMessage'][topic]) and return it using res.json().

Same holds true for publish. It accepts three parameters called brokerKey, topic and message, then correspondingly publishes the message on the broker. You may want to add custom logic here including authentication logic based on API keys, etc.

Starting the Express server

The final step is to start the Express server. Use a custom port, or fallback to a default if not provided.

const PORT = process.env.PORT || 3000;

app.listen(PORT, () => {
  console.log(`Server is running on http://localhost:${PORT}`);
});

This setup provides a reliable foundation for creating an MQTT-HTTP gateway. You can further enhance it with persistent storage, advanced authentication, or message processing workflows. Let me know how this implementation works for you—or share your own ideas for improvement—in the comments below. Happy coding!

Building a simple customer management system in PHP with MySQL

2024-11-20T00:00:00+00:00

Creating a customer management system (CMS) is a great way for beginners to learn PHP and MySQL. This hands-on project will guide you through the process of building a simple CMS from scratch, covering database design, CRUD operations, and form handling.

What is a customer management system?

A customer management system is a tool used to store and manage customer information. It allows businesses to efficiently track customer details, such as names, email addresses, and phone numbers, using basic CRUD (Create, Read, Update, Delete) operations.

In this tutorial, you’ll learn how to:

Design a database for storing customer information.
Build a user-friendly web interface for managing customers.
Implement CRUD functionality with PHP and MySQL.

Prerequisites

To follow this tutorial, you need:

A local server environment like XAMPP or WAMP (for Windows) or MAMP (for macOS).
Basic knowledge of HTML, PHP, and SQL.
A text editor such as VS Code, Sublime Text, or Notepad++.

Step 1: Setting up the project

1.1 Install and configure your server

Download and install XAMPP or your preferred local server.
Start Apache and MySQL from the control panel.

1.2 Create the project folder

Navigate to the htdocs directory in XAMPP and create a folder named customer_management.

Step 2: Designing the database

2.1 Create the database

Open phpMyAdmin by navigating to http://localhost/phpmyadmin.
Click New, and name the database customer_db.

2.2 Create the `customers` table

Run the following SQL query to create the customers table:

CREATE TABLE customers (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL,
    phone VARCHAR(15),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

This table has the following columns:

id: A unique identifier for each customer.
name: The customer’s name.
email: The customer’s email address.
phone: The customer’s phone number.
created_at: A timestamp of when the customer was added.

Step 3: Building the front-end

3.1 Creating the HTML layout

Create a file named index.php in the customer_management folder:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Customer Management</title>
    <link rel="stylesheet" href="style.css">
</head>
<body>
    <div class="container">
        <h1>Customer Management System</h1>
        <a href="add_customer.php" class="btn">Add New Customer</a>
        <table>
            <thead>
                <tr>
                    <th>ID</th>
                    <th>Name</th>
                    <th>Email</th>
                    <th>Phone</th>
                    <th>Actions</th>
                </tr>
            </thead>
            <tbody>
                <!-- Dynamic content will go here -->
            </tbody>
        </table>
    </div>
</body>
</html>

Step 4: Writing the back-end logic

4.1 Database connection

Create a file named db.php to handle database connectivity:

<?php
$host = 'localhost';
$user = 'root';
$password = '';
$database = 'customer_db';

$conn = new mysqli($host, $user, $password, $database);

if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}
?>

4.2 Fetch and display customers

Update index.php to fetch customer data dynamically:

<?php
include 'db.php';

$sql = "SELECT * FROM customers";
$result = $conn->query($sql);
?>

<tbody>
    <?php if ($result->num_rows > 0): ?>
        <?php while ($row = $result->fetch_assoc()): ?>
            <tr>
                <td><?= $row['id']; ?></td>
                <td><?= $row['name']; ?></td>
                <td><?= $row['email']; ?></td>
                <td><?= $row['phone']; ?></td>
                <td>
                    <a href="edit_customer.php?id=<?= $row['id']; ?>" class="btn-edit">Edit</a>
                    <a href="delete_customer.php?id=<?= $row['id']; ?>" class="btn-delete">Delete</a>
                </td>
            </tr>
        <?php endwhile; ?>
    <?php else: ?>
        <tr>
            <td colspan="5">No customers found.</td>
        </tr>
    <?php endif; ?>
</tbody>

4.3 Adding a customer

Create add_customer.php:

<?php
include 'db.php';

if ($_SERVER['REQUEST_METHOD'] == 'POST') {
    $name = $_POST['name'];
    $email = $_POST['email'];
    $phone = $_POST['phone'];

    $sql = "INSERT INTO customers (name, email, phone) VALUES ('$name', '$email', '$phone')";
    if ($conn->query($sql)) {
        header('Location: index.php');
    } else {
        echo "Error: " . $conn->error;
    }
}
?>

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Add Customer</title>
</head>
<body>
    <form method="POST" action="">
        <input type="text" name="name" placeholder="Name" required>
        <input type="email" name="email" placeholder="Email" required>
        <input type="text" name="phone" placeholder="Phone">
        <button type="submit">Add Customer</button>
    </form>
</body>
</html>

Step 5: Styling the app

Create a style.css file for basic styling:

body {
    font-family: Arial, sans-serif;
    background-color: #f9f9f9;
    color: #333;
}
.container {
    max-width: 800px;
    margin: 20px auto;
    padding: 20px;
    background: #fff;
    border: 1px solid #ddd;
    border-radius: 5px;
}
table {
    width: 100%;
    border-collapse: collapse;
    margin-top: 20px;
}
table th, table td {
    border: 1px solid #ddd;
    padding: 10px;
    text-align: left;
}
.btn {
    display: inline-block;
    padding: 8px 12px;
    color: white;
    background-color: #007bff;
    text-decoration: none;
    border-radius: 3px;
}

Step 6: Adding update and delete functionality

Update

Create edit_customer.php to edit customer details using a pre-filled form.

Delete

Create delete_customer.php to handle deleting a customer by ID.

Conclusion

By following these steps, you’ve built a fully functional customer management system using PHP and MySQL. This project taught you how to:

Design a database.
Implement CRUD operations.
Create a dynamic web interface.

You can now extend this project by adding features like search, sorting, or user authentication.

References

Unlocking Text from Embedded-Font PDFs: A pytesseract OCR Tutorial

2024-11-11T00:00:00+00:00

Extracting text from a PDF is usually straightforward when it’s in English and doesn’t have embedded fonts. However, once those assumptions are removed, it becomes challenging to use basic python libraries like pdfminer or pdfplumber. Last month, I was tasked with extracting text from a Gujarati-language PDF and importing data fields such as name, address, city, etc., into JSON format.

If the font is embedded in the PDF itself, simple copy-pasting won’t work, and using pdfplumber will return unreadable junk text. Therefore, I had to convert each PDF page to an image and then apply OCR using the pytesseract library to “scan” the page instead of just reading it. This tutorial will show you how to do just that.

Things you will need

pdfplumber (Python library)
pdf2image (Python library)
pytesseract (Python library)
tesseract-ocr

You can install the Python libraries using pip commands as shown below. For Tesseract-OCR, download and install the software from the official site. pytesseract is just a wrapper around the tesseract software.

pip install pdfplumber
pip install pdf2image
pip install pytesseract

Converting the PDF page to an image

The first step is to convert your PDF page to an image. This extract_text_from_pdf() function does exactly that-you pass the PDF path and the page_num (zero indexed) as parameters. Note that I’m converting the page to black and white first for clarity, this is optional.

# Extract text from a specific page of a PDF
def extract_text_from_pdf(pdf_path, page_num):
    # Use pdfplumber to open the PDF
    pdf = pdfplumber.open(pdf_path)
	print(f"extracting page {page_num}..")
	page = pdf.pages[page_num]
	images = convert_from_path(pdf_path, first_page=page_num+1, last_page=page_num+1)
	image = images[0]
	# Convert to black and white
	bw_image = convert_to_bw(image)
	# Save the B&W image for debugging (optional)
	#bw_image.save("bw_page.png")
	# Perform OCR on the B&W image
	e_text = ocr_image(bw_image)
	open('out.txt', 'w', encoding='utf-8').write(e_text)
	#print("output written to file.")
	try:
		process_text(page_num, e_text)
    except Exception as e:
        print("Error occurred:", e)
	print("done..")
	
# Convert image to black and white
def convert_to_bw(image):
    # Convert to grayscale
    gray = image.convert('L')
    # Apply threshold to convert to pure black and white
    bw = gray.point(lambda x: 0 if x < 128 else 255, '1')
    return bw
	
# Perform OCR using Tesseract on a given image
def ocr_image(image_path):
    try:
        # Perform OCR
        custom_config = r'--oem 3 --psm 6 -l guj+eng'
        text = pytesseract.image_to_string(image_path, config=custom_config)  # --psm 6 treats the image as a block of text
        return text
    except Exception as e:
        print(f"Error during OCR: {e}")
        return None

The ocr_image() function uses pytesseract to extract text from the image through OCR. The technical parameters like --oem and --psm control how the image is processed, and the -l guj+eng parameter sets the languages to be read. Since this PDF contained occasional English text, I used guj+eng.

Processing the text

Once you’ve imported the text using OCR, you can parse it in the format you want. This works similarly to other PDF libraries like pdfplumber or pypdf2.

nums = ['0', '૧', '૨', '૩', '૪', '૫', '૬', '૭', '૮', '૯']

def process_text(page_num, e_text):
    obj = None
    last_surname = None
    last_kramank = None
    print(f"processing page {page_num}..")
    for line in e_text.splitlines():
		line = line.replace('|', '').replace('[', '').replace(']', '')
        parts = [word for word in line.split(' ') if word]
        if len(parts) == 0: continue
        new_rec = True
        for char in parts[0]:
            if char not in nums:
                new_rec = False
                break
        if len(parts) < 2: continue
        
        if new_rec and len(parts[0]) >= 2: # numbered line
            if len(parts) < 9: continue
            if obj: records.append(obj)
            obj = {}
            last_surname = parts[1]
            obj['kramank'] = parts[0]
            last_kramank = parts[0]
            obj['full_name'] = ' '.join(parts[1:4])
            obj['surname'] = parts[1]
            obj['pdf_page_num'] = page_num + 1
            obj['registered_by'] = parts[4]
            obj['village_vatan'] = parts[5]
            obj['village_mosal'] = parts[6]
            if parts[8] == 'વર્ષ':
                idx = 7
                obj['dob'] = parts[idx] + ' વર્ષ'
                idx += 1
            elif len(parts[7]) == 8 and parts[7][2] == '-':
                idx = 7
                obj['dob'] = parts[idx]
            else:
                print("warning: no date")
                idx = 6
            obj['marital_status'] = parts[idx+1]
            obj['extra_fields'] = '::'.join(parts[idx+2:-2])
            obj['blood_group'] = parts[-1]
        elif parts[0] == last_surname: # new member in existing family
            if obj: records.append(obj)
            obj = {}
            obj['kramank'] = last_kramank
            obj['surname'] = last_surname
            obj['full_name'] = ' '.join(parts[0:3])
            obj['pdf_page_num'] = page_num + 1
            obj['registered_by'] = parts[3]
            obj['village_vatan'] = parts[4]
            obj['village_mosal'] = parts[5]
            if len(parts) <= 6: continue
            if parts[7] == 'વર્ષ': # date exists
                idx = 6
                obj['dob'] = parts[idx] + ' વર્ષ'
                idx += 1
            elif len(parts[6]) == 8 and parts[6][2] == '-':
                idx = 6
                obj['dob'] = parts[idx]
            else:
                print("warning: no date")
                idx = 5
            obj['marital_status'] = parts[idx+1]
            obj['extra_fields'] = '::'.join(parts[idx+2:-2])
            obj['blood_group'] = parts[-1]
        elif obj: # continuation lines
            if ("(" in line and ")" in line) or "મો.ઃ" in line:
                obj['extra_fields'] += ' ' + '::'.join(parts[0:])
    if obj: records.append(obj)        
    jstr = json.dumps(records, indent=4)
    open("guj.json", 'w', encoding='utf-8').write(jstr)
    print(f"written page {page_num} to json..")

Every PDF has its own nuances that must be accounted for. In this case, a new serial number (like 0૧ or 0૨) in the first field signaled a new group when the subsequent field (surname) changed.

pytesseract is a testament to the evolution and advancement in IT technology. About a decade ago, reading or parsing a PDF image using OCR in a non-English language on a modestly configured PC or laptop would have been nearly impossible. This is truly progress! Happy coding, and let me know how it goes in the comments below.

References

The indie blogger’s guide to seamless commenting: meet giscus

2024-11-09T00:00:00+00:00

Indie bloggers are on the constant lookout for tools that simplify their work while keeping costs low and user experience seamless. One of the most overlooked aspects of running a blog is finding the right comment hosting system. Enter giscus, a modern solution that stands out among its peers for being lightweight, privacy-friendly, and free to use. Here’s why it’s the best comment hosting system for the indie blogger living in today’s digital age.

Benefits of static hosting

No hassles of website maintenance

Indie bloggers love static websites for good reason. Unlike traditional dynamic websites, static sites come with zero server-side headaches. Once you’ve deployed your content, you don’t need to worry about database management, server updates, or unexpected downtime caused by backend failures. Platforms like Hugo, Jekyll, and Gatsby make it easier than ever to generate and maintain static sites. Add a robust version control system like GitHub, and your content stays safe with every update neatly tracked.

No costly backend bills to pay

Keeping a dynamic website up and running can be an expensive affair. Backend servers and database management come with their own set of recurring costs, from hosting fees to maintenance bills. These expenses can quickly add up, especially for indie bloggers who are not monetizing their content or are just starting out. Static site generators, paired with free hosting options, offer a refreshing alternative that doesn’t put a dent in your wallet.

Free platforms for the pleb

Gone are the days when hosting a blog meant shelling out money for a server. Platforms like GitHub Pages, Netlify, and Vercel offer free static site hosting that’s perfect for indie bloggers. These platforms come with seamless deployment pipelines, allowing you to push updates to your blog in a matter of seconds. And since static hosting doesn’t require server-side processing, the pages load lightning fast, ensuring a smooth user experience for readers.

The horrors of privacy-invasive third-party systems like Disqus

In the past, the go-to solution for adding comments to a blog was Disqus. While popular, Disqus became synonymous with intrusive ads, data tracking, and privacy concerns. Here are some of the pain points many bloggers and users have faced:

Tracking and privacy issues: Disqus is notorious for collecting user data and tracking user behavior across the web. For bloggers who value their readers’ privacy, this is a significant red flag.
Ads and monetization: Disqus’s free version often injects ads into your blog’s comment section, cluttering the user experience and taking away from the essence of the content.
Slow load times: Disqus can slow down the loading speed of your site due to its heavy scripts and external calls, which negatively impacts SEO and user satisfaction.

The modern blogger who champions simplicity and transparency needs an alternative that respects user privacy while being lightweight and functional.

The horrors of coding and maintaining your own commenting system

Some bloggers may consider building their own commenting system to avoid the pitfalls of third-party options. While this sounds like an appealing DIY project, it comes with its own set of nightmares:

Complexity: Developing a custom commenting system requires backend expertise, database integration, and server management. This can be overwhelming, especially for bloggers focused more on writing than on coding.
Security risks: Handling user-generated content means dealing with potential security threats like SQL injection and spam bots. Keeping the system secure requires ongoing updates and vigilance.
Maintenance: A custom-built system needs consistent maintenance. Any change in server configuration or software updates can disrupt the system and create additional work for the blogger.

For the indie blogger who doesn’t want to spend their time coding or worrying about security flaws, a more straightforward option is essential.

Enter the middle ground: why giscus is the best of both worlds

giscus offers a perfect balance between ease of use, privacy, and functionality. Here’s why it’s the ideal comment hosting system for the indie blogger:

Fully integrated with GitHub

giscus uses GitHub Discussions as its engine, so comments are stored as discussion threads in your repository. This means:

Transparent data management: All comment data is stored on GitHub, a platform that millions of developers trust.
Easy moderation: You can manage, edit, or delete comments directly from GitHub, leveraging the familiar interface and robust features.
Open-source spirit: giscus itself is open-source, which aligns with the values of transparency and collaboration many indie bloggers uphold.

Lightweight and fast

Unlike heavyweight comment systems, giscus is built with performance in mind. It doesn’t add bloat to your site, ensuring that your pages load quickly and smoothly. This is especially important for maintaining good SEO and providing a seamless experience for mobile readers.

Privacy-focused

giscus does not track users or display ads, making it an excellent choice for bloggers who want to maintain a privacy-friendly site. Since comments are facilitated through GitHub accounts, users already familiar with GitHub’s policies can engage confidently.

Simple installation

Setting up giscus is a breeze. Here’s how to get started:

Create a GitHub repository or use an existing one to store the discussions.
Visit the giscus website and generate the code snippet by selecting your repository and preferred discussion category.
Copy the provided code and paste it into your site’s codebase where you want the comment section to appear.
Deploy your site, and voilà! You have a fully functional, GitHub-based commenting system.

User-friendly for commenters

Engagement through giscus requires users to log in with their GitHub account, which appeals to a tech-savvy audience that values simplicity and privacy. For users already on GitHub, commenting feels seamless, and they can subscribe to discussions without having to create yet another account.

Community-driven enhancements

Because giscus is an open-source project, it evolves based on user feedback. Developers can contribute improvements, making it a community-driven tool that adapts over time.

Conclusion

Choosing the right comment system is critical for any blogger, especially those who want a balance between ease of use, privacy, and functionality. While legacy systems like Disqus have paved the way for commenting but introduced privacy issues, and coding your own system is often impractical, giscus shines as the modern, lightweight solution that respects privacy and integrates seamlessly with static sites. It’s time for indie bloggers to make the shift and experiment with giscus—the best comment hosting system for present times.

The emotional toll of Agile: burnout in the name of agility

2024-11-08T00:00:00+00:00

There is no doubt that over the last few years, Agile methodologies have gained prominence as a means to foster flexibility, innovation, and responsiveness to change. However, beneath the surface of this seemingly effective approach lies a troubling reality: the emotional toll it can take on teams. This article delves into the impact of Agile practices on team morale and well-being, particularly regarding constant iterations and pressure to deliver, exploring how the very principles intended to enhance productivity can inadvertently lead to burnout.

Understanding Agile: a double-edged sword

Agile practices prioritize iterative development, continuous feedback, and collaborative teamwork. On paper, these elements promote a dynamic work environment that encourages creativity and adaptability. Teams are empowered to experiment, pivot, and respond swiftly to customer needs. However, the relentless cycle of iterations, coupled with the expectation of constant delivery, can create a high-pressure atmosphere that strains team morale and emotional well-being.

The constant cycle of iteration

At the heart of Agile is the concept of iterative cycles—sprints that require teams to deliver incremental value within short timeframes. While this method enables quick responses to changing requirements, it also sets a relentless pace. Teams may find themselves caught in a continuous loop of planning, execution, review, and adjustment, leaving little room for reflection or recovery.

The emotional toll of this relentless cycle can manifest in several ways:

Increased Stress Levels: The pressure to meet tight deadlines and deliver high-quality work can lead to elevated stress levels among team members. The expectation of constant output can make individuals feel overwhelmed and anxious, particularly when faced with competing priorities.
Reduced Job Satisfaction: Continuous iterations may result in diminished job satisfaction. When team members feel they are perpetually “on the clock,” they may lose sight of the intrinsic value of their work, leading to disengagement and a decline in overall morale.
Erosion of Work-Life Balance: Agile’s focus on responsiveness can blur the lines between work and personal life. Team members may feel compelled to extend their work hours to keep up with demands, ultimately jeopardizing their work-life balance and well-being.

The pressure to deliver

The Agile philosophy encourages teams to prioritize customer needs and deliver value quickly. While this is essential for success, it can also create a culture of urgency that places undue pressure on team members. The fear of falling behind or disappointing stakeholders can lead to a toxic cycle of overwork and burnout.

Impacts of pressure on team morale

Fear of Failure: In an Agile environment, the fear of failure can be amplified by the fast-paced nature of work. Team members may become hesitant to take risks or voice concerns, fearing that any setback will reflect poorly on their performance.
Team Fragmentation: The pressure to deliver can lead to competition among team members rather than collaboration. When individuals are primarily focused on their own deliverables, the sense of teamwork and camaraderie may erode, resulting in a fragmented team dynamic.
Burnout: Perhaps the most concerning outcome of sustained pressure in an Agile context is burnout. Prolonged exposure to high-stress conditions can lead to physical and emotional exhaustion, cynicism, and a sense of inefficacy. This not only affects individual team members but can also have cascading effects on overall team productivity and performance.

Strategies for mitigating the emotional toll

Recognizing the potential for burnout in Agile teams is crucial for fostering a healthy work environment. Here are some strategies organizations can implement to mitigate the emotional toll:

Promote a Culture of Psychological Safety: Encouraging open communication and creating a safe space for team members to voice concerns can help alleviate the fear of failure. When individuals feel valued and heard, they are more likely to take risks and contribute meaningfully to the team.
Encourage Regular Breaks and Downtime: Organizations should emphasize the importance of work-life balance and encourage team members to take regular breaks. This can help individuals recharge and return to work with renewed energy and focus.
Set Realistic Expectations: While the Agile framework encourages responsiveness, organizations should strive to set realistic expectations for deliverables. Avoiding unrealistic deadlines and recognizing the limits of team capacity can help prevent burnout.
Foster Team Cohesion: Building strong interpersonal relationships among team members can enhance collaboration and support. Team-building activities, both in-person and virtual, can foster a sense of belonging and unity.
Conduct Regular Retrospectives: Regular retrospectives are a cornerstone of Agile practices. However, these sessions should focus not only on project outcomes but also on team well-being. Discussing emotional challenges and collectively brainstorming solutions can foster a supportive team environment.

Conclusion

While Agile methodologies offer numerous benefits for organizations, it is crucial to recognize the emotional toll they can exact on teams. The pressure to deliver, combined with the relentless cycle of iterations, can lead to burnout and negatively impact team morale. By prioritizing psychological safety, work-life balance, and realistic expectations, organizations can harness the full potential of Agile while safeguarding the well-being of their teams.

Ultimately, the goal should be to create a work environment where agility is not just about speed and efficiency but also about fostering a culture of well-being and collaboration—where teams can thrive without sacrificing their emotional health in the name of agility.

References

Beck, K., et al. (2001). Manifesto for Agile Software Development. Agile Alliance.
Goleman, D. (2013). Focus: The Hidden Driver of Excellence. HarperCollins.
Maslach, C., & Leiter, M. P. (2016). Burnout at Work: A Psychological Perspective. Psychology Press.
Sutherland, J., & Schwaber, K. (2017). The Scrum Guide: The Definitive Guide to Scrum: The Rules of the Game. Scrum.org.

Working with JSON data in PHP and MySQL: storing and retrieving complex structures

2024-11-04T00:00:00+00:00

In the ever-evolving world of web development, the ability to manage complex data structures efficiently is vital. JSON (JavaScript Object Notation) has become a standard for data exchange due to its lightweight format and ease of use. If you’re working with PHP and MySQL, understanding how to store and retrieve JSON data can greatly enhance the flexibility and capabilities of your applications. This guide will dive into using JSON columns in MySQL, interacting with them using PHP, and optimizing these processes.

Why use JSON in MySQL?

MySQL has supported JSON data types since version 5.7. This native support provides the ability to store complex, nested data structures without converting them to string formats. Here’s why JSON is beneficial in MySQL:

Ease of data modeling: JSON columns allow you to store arrays, objects, and other nested data structures without the need for additional tables or complex joins.
Native functions for manipulation: MySQL provides several JSON functions (JSON_EXTRACT, JSON_CONTAINS, JSON_SET, etc.) to query and manipulate data within JSON columns.
Performance improvements: JSON data is stored in a format that allows efficient parsing and retrieval, enhancing query performance compared to traditional string parsing.

Creating a JSON column in MySQL

To create a table with a JSON column, the syntax is straightforward:

CREATE TABLE user_data (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100),
    attributes JSON
);

The attributes column can store nested structures like:

{
    "preferences": {
        "theme": "dark",
        "notifications": true
    },
    "history": ["2024-11-01", "2024-11-02"]
}

Storing JSON data using PHP

PHP makes it simple to work with JSON through functions like json_encode() and json_decode(). When inserting data into a JSON column, you need to ensure the data is properly formatted. Here’s a practical example of how to insert data:

// Sample user data
$userAttributes = [
    "preferences" => [
        "theme" => "dark",
        "notifications" => true
    ],
    "history" => ["2024-11-01", "2024-11-02"]
];

// Convert the array to JSON
$jsonData = json_encode($userAttributes);

// Database connection (PDO)
$pdo = new PDO('mysql:host=localhost;dbname=testdb', 'username', 'password');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

// Insert JSON data
$stmt = $pdo->prepare("INSERT INTO user_data (name, attributes) VALUES (:name, :attributes)");
$stmt->execute(['name' => 'John Doe', 'attributes' => $jsonData]);

Retrieving and parsing JSON data

Fetching data from a JSON column in MySQL and decoding it in PHP is equally straightforward. Let’s retrieve and parse the stored data:

// Fetching data
$query = $pdo->query("SELECT attributes FROM user_data WHERE name = 'John Doe'");
$result = $query->fetch(PDO::FETCH_ASSOC);

if ($result) {
    $attributes = json_decode($result['attributes'], true);
    echo 'User Theme: ' . $attributes['preferences']['theme'];
}

The json_decode() function converts the JSON string into a PHP array when the true parameter is passed. This makes it easy to work with nested data directly.

Searching and filtering JSON data in MySQL

MySQL’s JSON functions allow you to perform searches and filter data based on specific JSON keys and values. For instance, you can use JSON_EXTRACT to access data or JSON_CONTAINS for searching.

Example: Retrieving users who have a dark theme preference

SELECT name FROM user_data
WHERE JSON_EXTRACT(attributes, '$.preferences.theme') = 'dark';

Updating JSON data in MySQL

Modifying JSON data directly in MySQL without replacing the entire content is a powerful feature. You can use the JSON_SET function to update specific keys.

Example: Updating a user’s notification setting

UPDATE user_data
SET attributes = JSON_SET(attributes, '$.preferences.notifications', false)
WHERE name = 'John Doe';

Best practices for working with JSON in PHP and MySQL

Validation: Always validate JSON data before storing or processing it. Use json_last_error() to check for any encoding/decoding issues in PHP.
Indexing: If you frequently query specific keys in your JSON data, consider using generated columns to index them for faster lookups.
Security: Sanitize any data before inserting it into your database to prevent SQL injection attacks.
Data structure considerations: JSON is ideal for semi-structured data but not for scenarios that require complex relational queries. Use traditional relational tables for such cases.

Advanced techniques: using generated columns for indexing

Generated columns can make querying JSON data more efficient by creating an indexable column derived from the JSON content.

ALTER TABLE user_data
ADD theme VARCHAR(20) AS (JSON_UNQUOTE(JSON_EXTRACT(attributes, '$.preferences.theme'))) STORED,
ADD INDEX (theme);

This allows you to query the theme column directly, improving performance:

SELECT name FROM user_data WHERE theme = 'dark';

Conclusion

Managing complex JSON structures in PHP and MySQL has become simpler with native JSON support. Whether you need to store user preferences, log histories, or any other semi-structured data, the combination of MySQL’s JSON data type and PHP’s JSON functions provides powerful and flexible solutions. Always remember to validate, secure, and optimize your data handling for the best performance and reliability.

Sources:

Handling large datasets in PHP: best practices for database management

2024-11-03T00:00:00+00:00

When dealing with vast amounts of data in PHP, the challenges are not just technical but strategic. Efficiently managing large datasets ensures that web applications remain fast, responsive, and user-friendly. In this guide, we’ll explore essential practices such as pagination, batch processing, and crafting efficient SQL queries.

Why handling large datasets is challenging

PHP’s strength lies in its versatility and ease of use. However, as datasets grow into millions of records, memory limitations and performance bottlenecks become significant concerns. Loading too much data at once can lead to high memory usage, sluggish response times, and even server crashes.

Best practices for managing large datasets

1. Use pagination

Pagination breaks down large result sets into smaller, more manageable pieces. By fetching only a subset of records per request, you optimize performance and provide a better user experience.

Example: Implementing pagination with LIMIT and OFFSET

$page = isset($_GET['page']) ? (int)$_GET['page'] : 1;
$records_per_page = 50;
$offset = ($page - 1) * $records_per_page;

$sql = "SELECT * FROM large_table LIMIT $records_per_page OFFSET $offset";
$result = $pdo->query($sql);

Advantages:

Reduces memory usage by limiting data retrieval.
Improves page load times and user navigation.

2. Implement batch processing

Batch processing handles data in smaller, sequential chunks, making large-scale operations more efficient and less resource-intensive. This technique is essential for data imports, exports, and complex data manipulations.

Example: Batch processing with a loop

$batch_size = 1000;
$total_records = 100000; // Assume we know this beforehand

for ($i = 0; $i < $total_records; $i += $batch_size) {
    $sql = "SELECT * FROM large_table LIMIT $batch_size OFFSET $i";
    $data_chunk = $pdo->query($sql);

    // Process data_chunk here
}

Applications:

Data migrations.
Processing log files or data transformations.

3. Optimize SQL queries

Writing efficient SQL queries is paramount when managing large datasets. Poorly optimized queries can drastically affect performance.

Key tips for query optimization:

Use indexes: Ensure that columns frequently used in WHERE, JOIN, or ORDER BY clauses are indexed.
Avoid SELECT *: Fetch only the columns you need to reduce data retrieval size.
Use LIMIT: Always limit the number of rows returned when fetching large data.
Leverage caching: Use PHP’s built-in caching mechanisms or third-party solutions like Redis to minimize database hits.

Example: Optimized query

$sql = "SELECT id, name, email FROM users WHERE status = 'active' ORDER BY created_at DESC LIMIT 50";

4. Use PHP’s PDO with buffered queries

Buffered queries load the complete result set into memory, allowing you to loop over data without blocking other queries. However, be cautious with this method for extremely large datasets as it can still consume significant memory.

Example: Using buffered queries

$stmt = $pdo->prepare("SELECT * FROM large_table");
$stmt->execute();

while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
    // Process each row
}

Memory management techniques

PHP’s memory can become a bottleneck when handling large data. Here’s how to manage it efficiently:

Increase PHP memory limit: Temporarily increase memory_limit in your script.
Unset variables: Free up memory by unsetting variables after use.
Garbage collection: Manually trigger garbage collection to reclaim memory.
```
unset($data_chunk);
gc_collect_cycles();
```

Leveraging external tools and databases

For highly scalable applications, integrating PHP with robust databases and tools is essential.

Use databases optimized for big data: Consider PostgreSQL or NoSQL databases like MongoDB when traditional RDBMS can’t handle your data efficiently.
Database sharding: Split large databases into smaller, manageable chunks across multiple servers.

Comparative table: PHP data management strategies

Method	Use Case	Pros	Cons
Pagination	Displaying user data, search results	Lightweight, user-friendly	Limited view at a time
Batch processing	Bulk imports/exports	Less resource-intensive	Higher code complexity
Buffered queries	Continuous data retrieval	Low memory consumption	Can block other processes
Indexing & optimized SQL	Data retrieval, sorting	Faster query execution	Can slow down inserts/updates
NoSQL databases	Real-time analytics, distributed systems	High scalability, flexibility	Learning curve, consistency trade-offs

Final thoughts

Handling large datasets in PHP is about balancing memory, processing speed, and efficient code. Implementing best practices like pagination, batch processing, and query optimization can greatly enhance application performance. As datasets continue to grow, staying updated on new PHP tools and methodologies will help you maintain efficiency.

Citations:

General best practices discussed in “High Performance MySQL” by O’Reilly Media.
PHP official documentation: PHP.net

Escaping the pseudo-innovation loop: why we need real progress, not just syntactic sugar

2024-11-02T00:00:00+00:00

In the rapidly evolving world of technology, we often celebrate every new tool, library, or framework as a breakthrough. Headlines scream about the “next big thing” in coding, and developers rush to adopt the latest trend, eager to stay ahead of the curve. But amidst this frenzied pace, an unsettling pattern has emerged: the cycle of pseudo-innovation. These are changes that add flash and novelty but contribute little more than superficial improvements—like syntactic sugar—to existing systems.

What is pseudo-innovation?

Pseudo-innovation refers to the type of technological change that, while marketed as a leap forward, essentially reworks existing ideas without substantial benefits. Think of it as the difference between repainting a room and remodeling an entire house. The former looks fresh but ultimately doesn’t change the structural function, while the latter transforms how the space is used.

Syntactic sugar, a term well-known in the developer community, encapsulates this idea perfectly. These are language features that make code easier to read or write but don’t fundamentally alter its behavior or capabilities. While these can be useful in moderation, when overused, they create a loop where progress appears to happen but, in reality, remains stagnant.

Examples of pseudo-innovation

One of the clearest manifestations of pseudo-innovation can be seen in the world of JavaScript frameworks. Not a year goes by without developers hearing about a new must-try tool that promises to revolutionize web development. For instance, when AngularJS emerged, it introduced a fresh way to build dynamic web applications. Yet soon after, the community saw an avalanche of alternatives like React, Vue, Svelte, and others. Each promised better syntax, a more streamlined approach, or a novel rendering technique, but how much of this is truly innovative?

Sure, React’s introduction of the virtual DOM was a significant technical shift, but the frameworks that followed largely focused on syntactical improvements or minor adjustments to developer experience rather than changing the capabilities of web applications themselves. The constant cycle of new libraries forces developers to spend more time learning the latest trends than creating substantive projects.

Table: Examples of pseudo-innovation in web development

Innovation Cycle	Examples of Syntactic Sugar	Resulting Effect
JavaScript frameworks	Hooks in React, Composition API in Vue	Makes code more readable, but doesn’t fundamentally change application capability
CSS advancements	New syntaxes like `@container` queries	Easier, but doesn’t revolutionize layout design
Language updates	Null coalescing in JavaScript (`??` operator)	Improves clarity but is not game-changing

The hidden cost of pseudo-innovation

The danger of pseudo-innovation is not just wasted time but a shift in focus. Developers become locked in a cycle of “learn, adapt, repeat,” prioritizing mastery over a new tool rather than addressing more meaningful challenges. When all efforts go toward following trends, less attention is given to tech that can solve real-world issues like climate change, educational inequality, or healthcare inefficiencies.

The obsession with being on the bleeding edge often stifles deep, long-term thinking. Why invest in groundbreaking environmental technologies when it’s easier to iterate on yet another JavaScript library? Why look for ways to develop inclusive educational platforms when most developers are busy debating over whether TypeScript is better than JavaScript?

What real innovation looks like

True innovation changes the rules of the game. It’s not just about improving aesthetics or making something slightly easier to write. Instead, it’s about addressing fundamental problems and creating value that goes beyond a better developer experience.

Examples of real innovation in action

Healthcare solutions: Real innovation can be found in AI tools that analyze medical scans to detect diseases like cancer early. This doesn’t just help specialists be more efficient—it saves lives.

Environmental tech: Breakthroughs in renewable energy, like more efficient solar panels or advancements in energy storage, represent genuine progress that impacts our global carbon footprint.

Education technology: Platforms like Khan Academy and Coursera have democratized learning, making education more accessible worldwide. Tools that leverage AI to provide personalized learning experiences have moved the needle in how education is perceived and accessed.

Comparison table: Real innovation vs. pseudo-innovation

Criteria	Pseudo-innovation	Real innovation
Scope	Cosmetic or syntactic changes	Fundamental shifts or new capabilities
Impact	Limited to developer productivity	Broad societal and industry impact
Sustainability	Often results in churn	Long-lasting, deeply rooted progress
Example	A new framework with different syntax	AI-powered diagnostic tools in healthcare

Avoiding the pseudo-innovation loop

Breaking free from the cycle of pseudo-innovation requires a change in perspective:

Focus on value, not novelty: The allure of shiny new tools is strong, but it’s crucial to ask, “What value does this add?” before investing time and resources.
Invest in long-term learning: Instead of cycling through the latest framework, developers can benefit more from mastering core concepts that transcend individual technologies—data structures, algorithms, and architectural patterns.
Encourage value-driven projects: As a tech community, we should celebrate innovations that have tangible benefits. Projects that improve efficiency, inclusivity, or sustainability deserve more spotlight than those that simply repackage old ideas with a modern twist.

Conclusion

In an industry obsessed with being “cutting edge,” it’s all too easy to confuse movement with progress. Syntactic sugar and pseudo-innovation may look appealing, but they often trap developers and companies in a loop where true impact is overlooked. By shifting our focus to meaningful, value-driven progress, we can ensure that our collective tech efforts go beyond incremental changes and contribute to solutions that matter. The future of technology shouldn’t just be easier to write—it should be transformative.

Optimizing PHP and MySQL for high-traffic websites

2024-11-01T00:00:00+00:00

Managing a high-traffic website effectively requires a robust and well-optimized backend to handle the surge of concurrent requests, minimize server load, and ensure a seamless user experience. PHP and MySQL, being a popular tech stack, offer various strategies and best practices to optimize performance. This guide explores key caching strategies, indexing techniques, query optimizations, and performance monitoring tools to help you scale PHP and MySQL efficiently.

1. Implementing caching strategies

Caching is one of the most effective ways to improve the performance of a PHP-MySQL web application. By storing frequently accessed data in a faster, intermediary storage layer, you can reduce the load on your database and accelerate response times.

Types of caching to consider:

Opcode caching: PHP can compile code to bytecode, which can be cached with tools like OPcache. This reduces the need for PHP to recompile scripts on every request, resulting in faster execution.
Data caching: Tools like Redis and Memcached store frequently accessed database query results. This approach is highly beneficial for read-heavy applications.
Page caching: Cache entire HTML pages for static content using systems like Varnish or built-in mechanisms in frameworks like Laravel.

Code example for Redis integration:

$redis = new Redis();
$redis->connect('127.0.0.1', 6379);
$cacheKey = 'user_data';

if ($redis->exists($cacheKey)) {
    $userData = $redis->get($cacheKey);
} else {
    $userData = fetchUserDataFromDB(); // Assume this fetches data from MySQL
    $redis->set($cacheKey, $userData, 3600); // Cache data for 1 hour
}

2. Using efficient indexing techniques

Indexes in MySQL can significantly enhance the speed of queries by allowing the database engine to find rows more quickly. However, improper or excessive indexing can lead to increased storage and maintenance overhead.

Best practices for indexing:

Use composite indexes when filtering queries with multiple columns. Ensure that columns with higher selectivity come first.
Avoid over-indexing, as maintaining too many indexes can slow down INSERT, UPDATE, and DELETE operations.
Analyze slow queries using MySQL’s EXPLAIN command to determine how your queries interact with indexes.

Example of creating a composite index:

CREATE INDEX idx_user_activity ON user_logs (user_id, action_type, timestamp);

3. Optimizing SQL queries

Inefficient SQL queries can lead to significant performance bottlenecks, especially under high traffic. Here’s how you can optimize your SQL:

Select only necessary columns: Avoid using SELECT * as it fetches unnecessary data.
Use LIMIT for pagination: When handling large datasets, use LIMIT and OFFSET to retrieve data in smaller chunks.
Avoid subqueries when possible: Replace subqueries with JOIN operations for better performance.
Normalize wisely: While normalization helps reduce redundancy, excessive normalization can lead to complex queries. Balance normalization with denormalization where necessary for read-heavy operations.

Optimized query example:

SELECT user_id, action_type FROM user_logs WHERE user_id = 101 AND timestamp > '2024-01-01' LIMIT 50;

4. Configuring MySQL for performance

Tuning MySQL settings to match your server’s resources and usage patterns can improve overall performance:

Adjust the innodb_buffer_pool_size to fit most of your database in memory. This is crucial for InnoDB storage engines.
Increase query_cache_size for scenarios with many identical queries, although caching engines like Redis are more effective.
Use slow_query_log to identify and optimize slow queries. This log helps pinpoint inefficient queries that take longer than a specified threshold.

Basic MySQL tuning snippet:

[mysqld]
innodb_buffer_pool_size = 2G
query_cache_size = 64M
slow_query_log = 1
slow_query_log_file = /var/log/mysql-slow.log
long_query_time = 2

5. Using PHP profiling and monitoring tools

Profiling and monitoring can help you identify bottlenecks in your PHP scripts and database interactions. Use these tools to track performance and make data-driven optimizations:

Xdebug: A PHP extension for in-depth profiling, showing function calls and execution time.
New Relic: Monitors server performance, application errors, and transaction traces in real-time.
Blackfire.io: Provides performance insights and recommendations for PHP scripts, ideal for identifying performance issues in complex applications.

Snippet for Xdebug setup:

[xdebug]
xdebug.mode=profile
xdebug.output_dir="/var/log/xdebug"

6. Load balancing and database sharding

For very high-traffic sites, scaling vertically may not be sufficient. Instead, use load balancing and database sharding to distribute traffic and reduce load on individual servers.

Load balancing: Distribute traffic across multiple PHP servers using tools like NGINX or HAProxy.

Database sharding: Partition your database into smaller, more manageable pieces based on logical data separation (e.g., sharding by user region).

Example of basic load balancing with NGINX:

upstream php_servers {
    server php1.example.com;
    server php2.example.com;
}

server {
    listen 80;
    location / {
        proxy_pass http://php_servers;
    }
}

Conclusion

Optimizing PHP and MySQL for high-traffic websites involves a multifaceted approach, from implementing effective caching and indexing to refining SQL queries and tuning server configurations. With the right strategies and tools, your application can handle large-scale traffic efficiently, ensuring a seamless experience for your users.

Here is the rewritten citation in a Wikipedia-style format:

References

MySQL Documentation Team, Oracle Corporation. MySQL Documentation: Optimization Guidelines.
PHP.net contributors. PHP Performance Best Practices.
Schwartz, Baron; Zaitsev, Peter; Tkachenko, Vadim. High Performance MySQL, 3rd Edition. O’Reilly Media, 2012.

From "My Computer" to "This PC": the evolution of language in a brand-centric consumer culture

2024-10-31T00:00:00+00:00

Language is a living, breathing phenomenon, evolving with society’s changing values, technological advancements, and cultural shifts. The transition from “My Computer” to “This PC” serves as a fascinating example of how linguistics have shifted toward branding and consumerism, rather than focusing on individual identity. This article dives deep into this evolution, exploring how language has transformed to reflect a more brand-centric and consumer-focused mindset, and the profound impact this shift has on our collective behavior and identity.

The personalization of language: a historical overview

Historically, language has been a powerful medium of expression, fostering connections among individuals. Words like “My Home” and “My Car” evoke personal significance, creating a sense of ownership and emotional attachment. These terms reinforce individuality and human connections, allowing people to express their experiences and identity.

The dawn of technology and consumer culture

However, with the rise of technology and a consumer-driven culture, a notable shift occurred. Terms that once embodied personal ownership began to reflect a standardized, commodified view of our interactions with products. For instance, the phrase “My Computer” once denoted a personal relationship with a device, but it gradually evolved into “This PC,” stripping away the personal touch in favor of a more generic term.

Table 1: The Shift in Language Over Time

Original Phrase	Evolved Phrase	Implication
My Computer	This PC	Shift from personal identity to product focus
My Phone	This Device	Move from personal connection to generic utility
My Profile	User Account	Commodification of personal online presence
My Music	Media Library	Transition from personal enjoyment to a curated collection

The rise of brand-centric language

Language as branding: examining “This PC”

The phrase “This PC” is more than just a change in terminology; it reflects a broader trend toward brand-centric language. The shift from “My” to “This” emphasizes the product rather than the individual user. This linguistic evolution aligns with the rise of branding as a powerful force that shapes consumer perception and behavior.

Implications for consumer identity

This transition highlights how language is tailored to appeal to a collective consumer identity rather than personal ownership. Companies like Microsoft promote this brand-centric terminology, fostering uniformity among users. Consequently, this shift diminishes individual differentiation while reinforcing brand loyalty.

Case studies in brand-centric language evolution

1. “My Phone” to “This Device”

Similar to the transition in computer terminology, smartphones have also undergone a linguistic evolution. The phrase “My Phone” once evoked personal connection, but “This Device” implies a more generic, product-focused view. This change encourages users to see their phones as mere tools for productivity rather than extensions of their identity.

2. “My Profile” to “User Account”

In the context of social media and online platforms, the terminology has shifted from “My Profile” to “User Account.” This evolution signifies a move toward a transactional view of online presence, highlighting the commodification of personal data. By prioritizing brand-centric language, platforms focus on users’ roles within a larger digital ecosystem rather than their individuality.

3. “My Music” to “Media Library”

The phrase “My Music” has transformed into “Media Library,” encapsulating the change from personal interests to a broader category of consumable media. This shift signifies the transition from personal enjoyment to a curated collection of commercially available content, reinforcing the brand’s role in defining user experiences.

The impact on collective mindset and behavior

Shaping consumer identity

Language plays a crucial role in shaping our collective identity as consumers. The prevalence of brand-centric terminology fosters a mindset focused on consumerism, where products and brands define individuals more than personal experiences or values. As consumers increasingly identify with brands rather than personal narratives, the language they use reflects this shift.

Identity formation

This linguistic evolution influences how individuals perceive themselves and their relationships with products. Phrases emphasizing brand identity can diminish personal agency, reinforcing a mindset that values material possessions over intrinsic human qualities.

The role of marketing and advertising

Marketing strategies wield significant influence over the language we use, reinforcing brand-centric narratives that guide individuals to adopt consumerist identities. Advertisements often employ language that promotes brand loyalty, subtly shifting our perceptions of self and identity.

Example: “Join the movement”

This phrase, frequently used in marketing campaigns, positions consumers as part of a larger community, suggesting that purchasing a product equates to participating in a significant cultural shift. This linguistic framing encourages consumers to see themselves not as individuals but as part of a collective identity shaped by brand affiliation.

The effects on our behavior and culture

Consumer behavior and decision-making

The evolution of language into a brand-centric approach has profound implications for consumer behavior. Language shapes our thoughts and perceptions, influencing how we make purchasing decisions. The more we engage with brand-centric language, the more likely we are to prioritize brand loyalty over personal preferences.

Emotional triggers: Marketers understand the power of language in eliciting emotional responses. Phrases like “limited edition,” “exclusive access,” or “unbeatable value” create urgency and exclusivity, motivating consumers to act quickly. This psychological manipulation showcases how language can drive consumer behavior, often at the expense of personal agency.

Cultural implications

The linguistic shift toward brand-centric language also impacts culture on a broader scale. As language reflects societal values, the prevalence of brand-centric terminology reinforces consumerism as a dominant cultural narrative. This transformation can lead to a society that prioritizes material possessions and brand affiliations over personal connections and experiences.

The dangers of consumer-centric language

As our language becomes increasingly focused on branding and consumerism, we risk losing sight of our individuality and shared human experiences. The more we engage with a language that promotes products over people, the more we distance ourselves from authentic connections.

The pushback: reclaiming personal language

In a world dominated by brand-centric language, there is a growing need to reclaim personal language that emphasizes individuality and authentic connection. Individuals and communities can challenge the prevailing narratives by prioritizing language that reflects personal values and experiences.

Advocacy for personal expression: Initiatives promoting self-expression and individuality can encourage people to reclaim their identities. Encouraging language that resonates with personal narratives can foster a sense of belonging and connection in an increasingly commercialized world.

Conclusion: the future of language and identity

The evolution of language from personal ownership to brand-centric terminology illustrates a significant shift in how we perceive ourselves and our relationships with the world. As language becomes more consumerism-centric, it is crucial to reflect on its impact on collective mindset, behavior, and identity.

Reclaiming personal language can empower individuals to reconnect with their identities beyond brands. By fostering a more people-centric approach to language, we can create a cultural environment that values individual experiences and narratives over commodified identities. The evolution of language serves as a powerful reminder of the importance of communication in shaping our perceptions of self, society, and the brands we engage with.

In this age of consumerism, it is vital to recognize the role of linguistics in shaping our collective consciousness and strive for a linguistic landscape that celebrates individuality alongside the brands that shape our world.

References

Crystal, D. (2008). A Dictionary of Linguistics and Phonetics. Wiley-Blackwell.
McLuhan, M. (1964). Understanding Media: The Extensions of Man. McGraw-Hill.
Postman, N. (1993). Technopoly: The Surrender of Culture to Technology. Knopf.

Creating a dynamic airline ticket booking system with PHP and MySQL

2024-10-31T00:00:00+00:00

In this in-depth guide, we’ll cover how to create a dynamic airline ticket booking system using PHP and MySQL. This system will allow users to search flights, view availability, book tickets, and manage bookings. We’ll also highlight best practices for building efficient reservation systems, including user authentication and secure data handling.

Introduction to the airline booking system
Project setup and prerequisites
Database structure and design
User authentication module
Flight search and availability check
Ticket booking and reservation management
Handling payments (optional integration)
Additional resources and open-source projects
Conclusion

1. Introduction to the airline booking system

An airline ticket booking system is a web application that enables users to check flight availability, book seats, and view their bookings. This type of system can be useful for travel agencies, airlines, and developers seeking to improve their skills with a real-world PHP project.

The primary components of the system will include:

User registration and login
Flight database and search functionality
Booking management and ticket reservation
Optional payment gateway integration for a complete e-commerce experience

This guide assumes a basic understanding of PHP, MySQL, and HTML.

2. Project setup and prerequisites

Environment setup:

PHP: Version 7.4 or higher.
MySQL: For storing user and flight data.
Apache or Nginx server: To serve the application locally.

Ensure that PHP, MySQL, and your preferred server are installed and configured correctly.

Project Structure:

airline-booking-system/
│
├── index.php                 # Home page for searching flights
├── register.php              # User registration
├── login.php                 # User login
├── flights.php               # Search flights page
├── book.php                  # Booking and reservation page
├── manage_booking.php        # Manage bookings for users
├── assets/                   # Folder for CSS, JavaScript
│   └── styles.css            # Basic styling for the application
└── db/
    └── database.sql          # SQL file to create necessary tables

3. Database structure and design

Designing an effective database schema is crucial for managing flights, reservations, and user data. Here is a simple schema that includes three main tables: users, flights, and bookings.

-- Table to store user data
CREATE TABLE users (
    id INT PRIMARY KEY AUTO_INCREMENT,
    name VARCHAR(100),
    email VARCHAR(100) UNIQUE,
    password VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Table to store flight information
CREATE TABLE flights (
    id INT PRIMARY KEY AUTO_INCREMENT,
    flight_number VARCHAR(20) UNIQUE,
    origin VARCHAR(100),
    destination VARCHAR(100),
    departure_time DATETIME,
    arrival_time DATETIME,
    seats_available INT,
    price DECIMAL(10, 2)
);

-- Table to store booking information
CREATE TABLE bookings (
    id INT PRIMARY KEY AUTO_INCREMENT,
    user_id INT,
    flight_id INT,
    booking_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    status ENUM('confirmed', 'pending', 'cancelled'),
    FOREIGN KEY (user_id) REFERENCES users(id),
    FOREIGN KEY (flight_id) REFERENCES flights(id)
);

users: Stores user data, including password hashes for secure authentication.
flights: Stores information about flights, availability, and pricing.
bookings: Links users to flights, allowing them to book and manage reservations.

4. User authentication module

To manage user registration and login, we’ll implement basic authentication. Start by creating a registration form (register.php) that collects user information and stores hashed passwords.

// register.php - User registration

if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    include 'db/connection.php';
    $name = $_POST['name'];
    $email = $_POST['email'];
    $password = password_hash($_POST['password'], PASSWORD_BCRYPT);

    $query = "INSERT INTO users (name, email, password) VALUES (?, ?, ?)";
    $stmt = $conn->prepare($query);
    $stmt->bind_param("sss", $name, $email, $password);
    $stmt->execute();
    $stmt->close();
}

For login (login.php), verify the email and password against stored values, starting a session if credentials are valid.

// login.php - User login

session_start();
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    include 'db/connection.php';
    $email = $_POST['email'];
    $password = $_POST['password'];

    $query = "SELECT * FROM users WHERE email = ?";
    $stmt = $conn->prepare($query);
    $stmt->bind_param("s", $email);
    $stmt->execute();
    $result = $stmt->get_result();
    $user = $result->fetch_assoc();

    if ($user && password_verify($password, $user['password'])) {
        $_SESSION['user_id'] = $user['id'];
        header("Location: index.php");
    } else {
        echo "Invalid email or password.";
    }
}

5. Flight search and availability check

On the main page (index.php), allow users to search for flights by origin, destination, and date. Use the flights table to retrieve results matching the user’s criteria.

// index.php - Search for flights

if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    include 'db/connection.php';
    $origin = $_POST['origin'];
    $destination = $_POST['destination'];
    $date = $_POST['date'];

    $query = "SELECT * FROM flights WHERE origin = ? AND destination = ? AND DATE(departure_time) = ?";
    $stmt = $conn->prepare($query);
    $stmt->bind_param("sss", $origin, $destination, $date);
    $stmt->execute();
    $flights = $stmt->get_result();
}

The flights.php (Search Flights Page) file is intended to allow users to search for flights by specifying criteria like origin, destination, and date, and then display available flights based on those inputs. Here’s how it could be implemented:

<!-- flights.php -->
<!DOCTYPE html>
<html>
<head>
    <title>Search Flights</title>
</head>
<body>
    <h1>Search for Flights</h1>
    <form method="POST" action="flights.php">
        <label for="origin">Origin:</label>
        <input type="text" name="origin" required>
        
        <label for="destination">Destination:</label>
        <input type="text" name="destination" required>
        
        <label for="date">Date:</label>
        <input type="date" name="date" required>
        
        <button type="submit">Search</button>
    </form>

    <?php
    if ($_SERVER['REQUEST_METHOD'] === 'POST') {
        include 'db/connection.php';
        $origin = $_POST['origin'];
        $destination = $_POST['destination'];
        $date = $_POST['date'];

        // Query to search flights
        $query = "SELECT * FROM flights WHERE origin = ? AND destination = ? AND DATE(departure_time) = ?";
        $stmt = $conn->prepare($query);
        $stmt->bind_param("sss", $origin, $destination, $date);
        $stmt->execute();
        $result = $stmt->get_result();

        echo "<h2>Available Flights</h2>";
        if ($result->num_rows > 0) {
            echo "<table><tr><th>Flight Number</th><th>Departure</th><th>Arrival</th><th>Seats</th><th>Price</th></tr>";
            while ($flight = $result->fetch_assoc()) {
                echo "<tr>
                    <td>{$flight['flight_number']}</td>
                    <td>{$flight['departure_time']}</td>
                    <td>{$flight['arrival_time']}</td>
                    <td>{$flight['seats_available']}</td>
                    <td>{$flight['price']}</td>
                    <td><a href='book.php?flight_id={$flight['id']}'>Book Now</a></td>
                </tr>";
            }
            echo "</table>";
        } else {
            echo "No flights found.";
        }
    }
    ?>
</body>
</html>

6. Ticket booking and reservation management

On the book.php page, allow users to book a flight, reducing the number of available seats. Update the bookings and flights tables to reflect the reservation.

// book.php - Book a flight

if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    include 'db/connection.php';
    $flight_id = $_POST['flight_id'];
    $user_id = $_SESSION['user_id'];

    // Check seat availability
    $query = "SELECT seats_available FROM flights WHERE id = ?";
    $stmt = $conn->prepare($query);
    $stmt->bind_param("i", $flight_id);
    $stmt->execute();
    $flight = $stmt->get_result()->fetch_assoc();

    if ($flight['seats_available'] > 0) {
        // Update seats and create booking
        $update_seats = "UPDATE flights SET seats_available = seats_available - 1 WHERE id = ?";
        $stmt = $conn->prepare($update_seats);
        $stmt->bind_param("i", $flight_id);
        $stmt->execute();

        $create_booking = "INSERT INTO bookings (user_id, flight_id, status) VALUES (?, ?, 'confirmed')";
        $stmt = $conn->prepare($create_booking);
        $stmt->bind_param("ii", $user_id, $flight_id);
        $stmt->execute();
        echo "Booking confirmed!";
    } else {
        echo "No seats available.";
    }
}

The manage_booking.php file allows users to view and manage their existing bookings, such as viewing current reservations, canceling them, or checking status. Here’s a basic implementation for managing bookings:

<!-- manage_booking.php -->
<?php
session_start();
if (!isset($_SESSION['user_id'])) {
    header("Location: login.php");
    exit;
}

include 'db/connection.php';
$user_id = $_SESSION['user_id'];

// Fetch bookings for the logged-in user
$query = "SELECT bookings.id, flights.flight_number, flights.origin, flights.destination, flights.departure_time, bookings.status
          FROM bookings
          JOIN flights ON bookings.flight_id = flights.id
          WHERE bookings.user_id = ?";
$stmt = $conn->prepare($query);
$stmt->bind_param("i", $user_id);
$stmt->execute();
$result = $stmt->get_result();
?>

<!DOCTYPE html>
<html>
<head>
    <title>Manage Bookings</title>
</head>
<body>
    <h1>Your Bookings</h1>
    <?php if ($result->num_rows > 0): ?>
        <table>
            <tr>
                <th>Flight Number</th>
                <th>Origin</th>
                <th>Destination</th>
                <th>Departure</th>
                <th>Status</th>
                <th>Actions</th>
            </tr>
            <?php while ($booking = $result->fetch_assoc()): ?>
                <tr>
                    <td><?php echo $booking['flight_number']; ?></td>
                    <td><?php echo $booking['origin']; ?></td>
                    <td><?php echo $booking['destination']; ?></td>
                    <td><?php echo $booking['departure_time']; ?></td>
                    <td><?php echo ucfirst($booking['status']); ?></td>
                    <td>
                        <?php if ($booking['status'] === 'confirmed'): ?>
                            <form method="POST" action="cancel_booking.php">
                                <input type="hidden" name="booking_id" value="<?php echo $booking['id']; ?>">
                                <button type="submit">Cancel</button>
                            </form>
                        <?php endif; ?>
                    </td>
                </tr>
            <?php endwhile; ?>
        </table>
    <?php else: ?>
        <p>No bookings found.</p>
    <?php endif; ?>
</body>
</html>

Similarly, the cancel_booking.php could handle booking cancellation logic, updating the booking status in the bookings table when a user requests to cancel. The implementation of this file is left to the user as a home-work exercise.

7. Handling payments (optional integration)

For a full-featured booking system, consider adding a payment module. Popular options include PayPal, Stripe, and Razorpay. Many offer PHP SDKs for easy integration. Implementing payment handling will require a new payments table and modifications to the booking process.

8. Additional resources and open-source projects

For further learning and to expand your project, consider checking out these open-source repositories and resources:

PHP-MySQL Flight Booking System on GitHub (Add actual relevant repositories here)
Official PHP documentation on MySQLi and PDO
Tutorials on handling authentication and authorization in PHP

9. Conclusion

Creating a dynamic airline booking system in PHP and MySQL is a rewarding project that offers insights into both backend development and relational database management. By following this guide, you should now have a working ticket reservation system, from database design to user authentication and booking management.

The myth of one-size-fits-all: why standard ERP systems often fail

2024-10-30T00:00:00+00:00

Enterprise Resource Planning (ERP) systems are the backbone of countless organizations, promising seamless integration of business processes, improved efficiency, and centralized data management. Yet, the glitter of ERP solutions sometimes conceals a harsh reality: the promise of “one-size-fits-all” is often a myth. While ERP systems claim to be universal solutions, the truth is they frequently struggle to fit the unique needs of specific industries or individual businesses. Let’s explore why these supposedly standardized ERP systems often fail and the costly customizations many companies must endure to make them work.

The allure of the standardized ERP

For many organizations, the idea of a standardized ERP system is tempting. It offers a tidy package of integrated modules to manage everything from accounting and inventory to sales and human resources. Theoretically, it’s supposed to eliminate data silos, automate tasks, and enhance decision-making by providing real-time insights. Companies are attracted to the idea of a universal, off-the-shelf solution that reduces complexity and is easy to implement.

However, this narrative overlooks a crucial aspect of business: no two organizations are alike. Even companies within the same industry can vary significantly in their operations, workflows, and requirements. And here lies the primary flaw of the standardized ERP approach.

Why a generic ERP often falls short

When businesses buy a standardized ERP solution, they are essentially purchasing a mold—a pre-designed framework that presumes to know what a “typical” company needs. However, reality paints a different picture. Each organization has its own legacy processes, market demands, customer expectations, and unique value propositions. These are just a few of the reasons why a generic, one-size-fits-all ERP often falls short:

Industry-specific needs: Every industry has distinct regulations, standards, and processes. A retail chain’s ERP must prioritize inventory management and customer relationships, while a manufacturing company might need a heavy focus on supply chain management and production tracking. Standardized ERPs struggle to account for these differences, leading to a mismatch between what the software offers and what the company needs.
Unique workflows and processes: Businesses often build competitive advantages around specific workflows. For example, a company might have a unique sales process that distinguishes it from competitors. Standard ERP systems are designed for generic workflows and may not align with these unique practices, leading to inefficient workarounds or forced changes in business processes.
Cultural and geographical differences: Organizations operating globally face diverse requirements regarding taxation, language, legal compliance, and cultural preferences. A standardized ERP may overlook these regional nuances, leading to operational and compliance issues in different branches.
Scalability and flexibility: Businesses evolve over time, and their ERP systems need to scale and adapt to changing demands. Standard ERP systems often have rigid structures that limit scalability or require extensive reconfigurations, making them less flexible for organizations in growth phases or those pivoting in their strategies.

The high cost of customization

When a company realizes its standardized ERP isn’t meeting its needs, the next step is customization. While customizations are a necessary evil to achieve a better fit, they often come with hidden costs and complexities.

Time and budget overruns: Customizing an ERP system can turn into a long and expensive process. Modifying the core functions, tweaking workflows, and integrating new modules all require additional time and budget, which can escalate beyond initial expectations. According to research by Panorama Consulting, more than half of ERP implementations end up over budget due to unforeseen costs—often due to extensive customizations.
Increased maintenance complexity: Customizations can complicate the software’s architecture. Future updates, patches, or integrations with third-party systems can become cumbersome, requiring businesses to continually rework custom code or manually test new features. The complexity only grows as the ERP becomes more deeply embedded in the company’s ecosystem.
Loss of vendor support: Many ERP vendors hesitate to fully support highly customized systems. Custom code might not be compatible with standard updates or improvements, leaving businesses in a vulnerable position when bugs or system errors arise. This lack of support can lead to more downtime, decreased productivity, and higher costs for ongoing maintenance.
Compromised user experience: Customizations add complexity not just in maintenance but also in usage. Employees accustomed to a certain workflow or process may find it challenging to adapt to new, custom-designed ERP modules that don’t fit seamlessly into their established routines.

Real-world examples: Where standardized ERPs failed

There are numerous high-profile examples of standardized ERP implementations gone awry. A well-known instance is that of Hershey’s ERP failure in 1999. The candy giant attempted to implement an ERP system to streamline its supply chain and order processes. However, due to customization issues, it ended up with massive problems in its distribution, leading to a loss of $100 million in sales during the Halloween season—its most crucial sales period.

Another example is Lidl’s SAP disaster, where the German retailer invested seven years and €500 million in a new SAP-based ERP system, only to abandon the project because the system couldn’t handle Lidl’s custom pricing structure. Despite SAP being a powerful system, it was too rigid to accommodate Lidl’s unique pricing and inventory practices.

The fallacy of one-size-fits-all: Why industry-specific ERPs may be the solution

Many ERP vendors are beginning to recognize that their standardized solutions are insufficient for companies in highly specialized industries. As a result, industry-specific ERP systems are gaining traction. These systems offer pre-built modules tailored to the requirements of certain verticals, such as healthcare, retail, construction, or manufacturing.

For example, Epicor offers ERP solutions with modules specifically designed for manufacturing companies, addressing their needs in areas like supply chain management, job scheduling, and production quality control. Meanwhile, Cerner focuses on ERP systems for the healthcare industry, with features that prioritize patient records, billing, and compliance with healthcare regulations.

Making the right ERP choice

Before diving headfirst into ERP implementation, it’s crucial for businesses to conduct a thorough needs assessment and evaluate their unique processes. Here are some recommendations:

Understand your key requirements: Identify which business processes are critical and how the ERP should support them. List the specific features that your organization cannot function without.
Prioritize industry-specific solutions: Explore ERP systems tailored to your industry. Such solutions typically come with built-in features for specialized processes and compliance requirements.
Plan for flexibility: Choose an ERP that allows you to configure without significant code changes. This ensures smoother future updates and reduces ongoing costs.
Involve stakeholders from the start: Ensure that key employees, particularly those who will use the ERP daily, are part of the decision-making process. Their insights into existing workflows can help avoid implementation headaches down the road.

Conclusion

The allure of a one-size-fits-all ERP is strong, but it’s crucial to remember that businesses are as diverse as the industries they serve. The promise of a universal ERP solution often ignores the complexities of real-world operations, leading to customization headaches, blown budgets, and user dissatisfaction. By carefully assessing your needs, prioritizing industry-specific solutions, and staying wary of rigid systems, you can avoid the common pitfalls of generic ERP systems and ensure your organization’s success.

When it comes to ERP solutions, remember: if the shoe doesn’t fit, it’s not your foot that needs changing—it’s the shoe. The myth of one-size-fits-all is just that—a myth. So, find the ERP that fits your organization, not the other way around.

References:

Panorama Consulting Solutions, “2023 ERP Report: Key Findings on ERP Implementations.”
ERP Focus, “ERP Failures: Real Life Stories.”
Gartner, “Critical Capabilities for ERP in 2024.”

The illusion of simplicity: how OOP can overcomplicate simple problems

2024-10-29T00:00:00+00:00

Object-oriented programming (OOP) has been hailed as the savior of the software world, promising more manageable codebases and scalable applications. From encapsulation to inheritance, the paradigm offers a toolkit that is designed to make developers’ lives easier. Yet, over the years, OOP has gained its fair share of critics who argue that the principles it upholds can often lead to overcomplication—especially when applied to simple problems. So, is OOP truly the answer to every software challenge, or is it an illusion of simplicity, cloaking unnecessary complexity beneath a facade of best practices?

In this deep dive, we’ll explore why the abstraction and encapsulation principles of OOP might be overkill for simple problems and when sticking to basic solutions or alternative paradigms can lead to more effective and maintainable code.

The promise of object-oriented programming: simplicity through abstraction

The core philosophy of OOP is to model software as a collection of objects that interact with each other. It’s akin to representing the real world in code. Principles like inheritance, polymorphism, and encapsulation provide a way to make code modular, reusable, and maintainable. In theory, this sounds like an absolute win.

Consider an example of a basic Car class:

class Car {
    private String color;
    private int speed;

    public Car(String color, int speed) {
        this.color = color;
        this.speed = speed;
    }

    public void drive() {
        System.out.println("The car is driving at " + speed + " km/h.");
    }

    public void repaint(String newColor) {
        this.color = newColor;
    }
}

On the surface, it looks quite simple and intuitive. If you want to create a car, you instantiate an object with a color and speed, and you can make it drive or repaint it with a method. It’s elegant—until complexity starts creeping in.

When simple tasks become complex

The illusion of simplicity in OOP begins to unravel when basic requirements evolve. Let’s say we need to support a new vehicle type: a truck. Following OOP principles, you may introduce a Truck class that inherits from Car, or you might abstract both into a common superclass called Vehicle. It looks like a small addition, but such minor changes snowball over time.

Layers of abstraction: How complexity multiplies

In larger OOP-based systems, creating classes for every concept or behavior quickly results in a deep inheritance hierarchy. A simple problem, like managing vehicles, might require dozens of classes and interfaces due to enforced abstraction.

For instance, to accommodate more types like electric cars and motorcycles, you might structure a hierarchy like this:

Vehicle (superclass)
- Car (inherits from Vehicle)
  - ElectricCar (inherits from Car)
- Truck (inherits from Vehicle)
- Motorcycle (inherits from Vehicle)

If you now need to add a behavior like calculateMaintenanceCost(), you could end up writing it in multiple places or creating an additional abstraction to handle it.

Encapsulation and its trade-offs

Encapsulation is another tenet of OOP that often contributes to this complexity. While encapsulation helps in hiding internal states and only exposing necessary details, enforcing this for simple problems can create a tangled mess of getters, setters, and boilerplate methods.

For instance, if the internal state of an object is simple, having strict getters and setters can lead to code that is difficult to refactor. It also creates an illusion that these methods are somehow necessary for safety or maintainability when the underlying data could be as simple as a couple of fields.

This proliferation of unnecessary boilerplate has even led to entire frameworks (like JavaBeans or Lombok in Java) existing solely to reduce the drudgery of writing these repetitive pieces of code. And at that point, you need to ask: Are we truly simplifying things?

Why OOP may not be the best fit for simple problems

Misalignment with the problem domain

In many cases, using an OOP approach to solve a simple problem can feel like over-engineering. If all you need is a script that calculates the sum of integers from 1 to 1000, writing a class named IntegerSummationService with multiple methods for “abstraction” is absurdly unnecessary.

Functional paradigms: Less fluff, more focus

Functional programming (FP) has gained traction as an alternative to OOP precisely because of its emphasis on simplicity and immutability. FP offers the flexibility of defining isolated functions, leading to more straightforward code for many straightforward problems.

For instance, consider how a vehicle maintenance cost might be handled in a functional style:

def calculate_maintenance(vehicle_type, age):
    if vehicle_type == 'car':
        return 200 + (age * 10)
    elif vehicle_type == 'truck':
        return 500 + (age * 20)
    else:
        return 100 + (age * 5)

No need for a sprawling class hierarchy; the logic is contained within a small, easy-to-understand function.

Procedural simplicity for simple needs

Sometimes, basic procedural programming—writing simple scripts and functions—can be more than adequate. For tasks like file reading, processing CSVs, or generating reports, there’s often no tangible benefit to wrapping the logic in classes or objects. The code remains more readable and easier to maintain without the OOP overhead.

The pitfalls of enforced best practices

The notion that OOP is the only way to organize code is a fallacy that often leads developers to overthink basic solutions. A classic example of this can be seen in frameworks that enforce design patterns, such as Model-View-Controller (MVC). While MVC is great for complex applications, forcing it on a simple CRUD application can lead to unnecessary classes, interfaces, and abstract factories.

It’s like forcing every home cook to set up a professional kitchen just to make a bowl of cereal.

An anecdote: The battle with classes and interfaces

Here’s a story that many developers can relate to. I once worked on a project where a senior developer insisted that every single domain object needed an accompanying factory class. For example, to instantiate an order, we had to go through an OrderFactory. This was justified as a best practice, and when I inquired about what it achieved, the answer was always the same: “It’s good design.”

What it ended up achieving was hundreds of lines of boilerplate code and confused junior developers who spent more time understanding factories than solving actual business problems.

When OOP works, and when it doesn’t

Of course, there are situations where OOP shines. For large-scale systems with numerous entities and relationships—such as banking systems, e-commerce platforms, or content management systems—OOP can help manage the complexity effectively. The benefits of encapsulation, polymorphism, and inheritance pay off in these scenarios.

However, using the same hammer for every nail isn’t always the right strategy. For smaller projects or simple tasks, other paradigms like functional programming, scripting, or procedural code can be more straightforward, easier to maintain, and require less boilerplate.

The key takeaway: balance over zealotry

The takeaway here is not to discard OOP altogether. It’s a valuable tool, but one that needs to be wielded judiciously. Treating it as a panacea can lead to an illusion of simplicity that hides layers of unnecessary complexity. As developers, we must remain adaptable, choosing the right paradigm based on the problem at hand rather than blindly adhering to OOP principles.

The next time you’re about to design a new system or refactor an existing one, ask yourself: Is OOP truly the simplest solution?

Purpose-driven technology: infusing utilitarianism back into innovation

2024-10-28T00:00:00+00:00

In an age dominated by rapid technological advancements, it’s easy to forget the original purpose behind the creation of technology: to enhance human potential, address pressing societal challenges, and drive progress. As we forge ahead, however, there’s a growing concern that innovation is increasingly motivated by novelty rather than utility. This article explores the notion of purpose-driven technology and the importance of reinstating utilitarian principles in our approach to innovation.

The roots of technology: utilitarianism in action

From the wheel to the internet, technology has historically been rooted in utilitarianism—the philosophy that emphasizes actions that produce the greatest benefit for the most people. Early inventors and engineers approached problems with a mindset focused on solving specific issues: agricultural tools improved food production, transportation systems connected communities, and medical advancements enhanced healthcare. These innovations were not mere artifacts of creativity; they were designed to meet real-world needs, improving lives and advancing society.

However, as technology evolved, the landscape shifted. The advent of the digital age ushered in a new era of innovation characterized by rapid development cycles, the proliferation of frameworks, libraries, and APIs, and an unrelenting push to create the next big thing. In this environment, it’s increasingly common to see technologies developed not for their utility, but simply for the sake of innovation. This shift raises crucial questions: Have we lost sight of the utilitarian goals that once guided technological progress? Are we prioritizing the creation of shiny new tools over their actual application in solving real-world problems?

The drift towards trivial innovation

In recent years, numerous examples highlight how technology has drifted away from its utilitarian roots.

1. Framework fatigue

Consider the world of web development, where new frameworks emerge at breakneck speed. Developers often find themselves in a constant cycle of learning and adopting the latest technologies—be it Angular, React, or Vue.js. While these frameworks offer unique features and functionalities, the relentless pursuit of the “next best thing” can lead to a fragmented ecosystem where projects become overly complex, hindering collaboration and efficiency.

Table 1: Comparison of Popular JavaScript Frameworks

Framework	Year Released	Main Purpose	Notable Features
Angular	2010	Single-page applications	Two-way data binding, dependency injection
React	2013	UI component development	Virtual DOM, component-based architecture
Vue.js	2014	Flexible and incremental adoption	Reactive data binding, easy integration

As a result, many developers find themselves using these frameworks without fully understanding their real-world applications, leading to projects that prioritize technical sophistication over tangible benefits.

2. The app overload

The app ecosystem presents another glaring example of trivial innovation. While the explosion of mobile applications has transformed how we interact with technology, it has also led to an overwhelming number of apps that often lack clear utility. Many apps are developed merely to jump on trends or capitalize on user behavior rather than genuinely addressing users’ needs.

For instance, countless “lifestyle” apps provide minimal value, often overlapping in functionality. Users are bombarded with choices, yet many struggle to find applications that truly enhance their daily lives. This saturation can lead to app fatigue, where the novelty of new apps quickly fades, leaving users disenchanted.

Realigning technology with utilitarian goals

To combat this trend, it’s essential to realign technology development with utilitarian principles, ensuring that innovation is driven by genuine human needs rather than superficial trends. Here are some strategies to foster purpose-driven technology:

1. Focus on problem-solving

Technology should be developed with a clear understanding of the problems it aims to solve. Developers and engineers need to engage with end-users to identify their pain points and collaborate on solutions that directly address those issues. By prioritizing utility over novelty, we can create products that make a meaningful impact on people’s lives.

2. Adopt a minimalist approach

In a world rife with complexity, embracing simplicity can be a powerful tool for innovation. By stripping away unnecessary features and focusing on core functionalities, developers can create more intuitive and effective solutions. This minimalist approach can lead to a better user experience and greater satisfaction.

3. Encourage interdisciplinary collaboration

The most successful innovations often arise from collaboration across disciplines. By fostering partnerships between technologists, social scientists, and domain experts, we can gain a more holistic understanding of the challenges facing society and create solutions that are both effective and relevant. Interdisciplinary teams can bridge the gap between technical capabilities and real-world applications, ensuring that innovations are rooted in utility.

As we develop new technologies, it’s crucial to consider their long-term impact on society and the environment. Prioritizing sustainability can guide innovation toward solutions that benefit not only the present but also future generations. By focusing on social impact, developers can create technologies that empower communities, enhance quality of life, and address pressing global issues, such as climate change, poverty, and inequality.

Conclusion: A call for purpose-driven innovation

The time has come to reinvigorate the utilitarian principles that once drove technological innovation. By shifting our focus back to problem-solving and genuine human welfare, we can cultivate a landscape of technology that enriches lives rather than merely filling the market with superficial trends.

Purpose-driven technology has the potential to transform society for the better, advancing human potential and addressing the challenges we face today. As developers, engineers, and innovators, we must ask ourselves: Are we creating technology that truly serves a purpose? By aligning our efforts with the needs of society, we can ensure that innovation remains a powerful force for good, paving the way for a brighter, more sustainable future.

Understanding the limitations of Agile: when flexibility becomes chaos

2024-10-27T00:00:00+00:00

Agile methodology, lauded for its adaptability and responsiveness, has revolutionized the way software development teams operate. However, beneath the surface of its acclaimed flexibility lies a paradox: the very principles that enable Agile’s success can also give rise to chaos when not properly managed. This article delves into the limitations of Agile, examining how its inherent flexibility can sometimes lead to confusion and misalignment, ultimately jeopardizing project outcomes.

The Agile promise: flexibility and responsiveness

At its core, Agile is designed to embrace change. The Agile Manifesto emphasizes collaboration, customer feedback, and iterative progress, allowing teams to pivot as requirements evolve. This promise of flexibility is undoubtedly appealing in today’s fast-paced, ever-changing technological landscape. However, this flexibility can become a double-edged sword.

The dangers of unchecked flexibility

Scope creep and shifting priorities
In a bid to remain responsive to customer needs, Agile teams may fall victim to scope creep. With every iteration, stakeholders might introduce new features or modify existing ones based on feedback, resulting in a project that deviates significantly from its original objectives. This continual shifting of priorities can lead to confusion among team members, as they struggle to align their efforts with an ever-changing roadmap.

Example: Imagine a development team working on a project management tool. Initially, the goal was to develop a user-friendly interface for task tracking. However, after multiple feedback sessions, stakeholders keep adding features like time tracking, project budgeting, and collaboration tools. Each new request stretches the team’s resources thin, leading to a product that is bloated and lacks cohesion.
Miscommunication and role ambiguity
Agile’s emphasis on self-organizing teams can inadvertently create role ambiguity. When team members have the freedom to take on various tasks, lines between responsibilities can blur. This lack of clarity may lead to miscommunication, misunderstandings, and ultimately a fragmented approach to project execution.

Example: In a Scrum team, if there is no clear definition of roles, developers might assume that testers will also handle bug fixes, while testers expect developers to fix the issues they identify. This overlap can result in critical bugs slipping through the cracks, causing frustration for both team members and stakeholders.
Reduced focus on long-term goals
Agile promotes short-term iterations and frequent releases, which can lead teams to prioritize immediate gains over long-term objectives. While delivering incremental value is essential, losing sight of the bigger picture can hinder a project’s success. Teams may become so engrossed in daily stand-ups and sprint reviews that they forget to align their work with the overall vision.

Example: A marketing automation platform may release updates every two weeks, focusing on user interface enhancements based on user feedback. However, without a strategic plan for integrating these features into a cohesive marketing strategy, the product may fail to meet user expectations in the long run, resulting in declining user engagement.

When Agile turns into chaos

The cumulative effect of these pitfalls can lead to chaos within an Agile framework. Here are some scenarios where flexibility transforms into disorder:

Overwhelmed teams
As the demands on an Agile team increase, members may feel overwhelmed by the influx of changes and requests. The pressure to adapt quickly can lead to burnout, low morale, and diminished productivity. When team members are constantly firefighting to keep up with shifting priorities, the quality of their work may suffer.
Stakeholder frustration
Stakeholders expect a clear, consistent communication flow and reliable progress updates. When an Agile team struggles to keep everyone aligned due to changing requirements, stakeholders may become frustrated and lose confidence in the team’s ability to deliver results. This erodes trust and can hinder collaboration.
Compromised product quality
The focus on flexibility can sometimes lead to corners being cut in terms of quality assurance. As deadlines loom and pressure mounts, teams may prioritize speed over thorough testing. This can result in a product that, while completed quickly, is riddled with bugs and usability issues.

Finding balance: establishing boundaries within Agile

To mitigate the chaos that can arise from Agile’s inherent flexibility, organizations must establish clear boundaries and practices:

Define a clear project scope
While flexibility is essential, establishing a baseline project scope at the outset can help maintain focus. This scope should be agreed upon by all stakeholders, with an emphasis on core features that align with user needs. Regular reviews can allow for adjustments while keeping the overall objectives intact.
Clarify roles and responsibilities
Ensuring that every team member understands their role within the Agile framework is crucial for effective communication and collaboration. Regularly revisiting role definitions during retrospectives can help keep everyone accountable and aligned.
Prioritize quality assurance
Integrating robust testing practices within the Agile process can help uphold product quality. Implementing automated testing and involving quality assurance from the beginning can catch issues early and maintain high standards.
Foster open communication
Encouraging transparent and open communication among team members and stakeholders can help mitigate misunderstandings. Utilizing collaboration tools can facilitate information sharing and keep everyone on the same page.

Conclusion

Agile methodologies hold immense potential for enhancing project outcomes through flexibility and responsiveness. However, without careful management, this very flexibility can spiral into chaos, resulting in confusion, burnout, and compromised product quality. By understanding the limitations of Agile and establishing clear boundaries, organizations can harness its power while avoiding the pitfalls that threaten to derail their projects. In the end, the goal should be to strike a balance between agility and structure, ensuring that teams remain aligned, focused, and productive.

Think async: unleashing PHP's hidden performance power

2024-10-25T00:00:00+00:00

Asynchronous programming is an often misunderstood yet crucial concept in modern software development. While traditionally associated with JavaScript or Python, asynchronous programming is just as powerful and relevant in PHP. In this guide, we’ll explore the foundations of asynchronous programming, why it matters, and how PHP developers can use it to build faster and more efficient applications.

Understanding synchronous vs. asynchronous programming

Before we dive into the nuts and bolts, let’s define two core concepts: synchronous and asynchronous programming.

Synchronous programming is how PHP (and many languages) typically handle code execution. Here, each line of code is executed one after the other. If a particular line involves a time-consuming operation, such as fetching data from a database or making an API call, the entire script waits until that operation completes. Think of it as a line at a coffee shop where customers are served one by one, and nobody else can be served until the current customer is done.

On the other hand, asynchronous programming allows multiple tasks to run simultaneously or at least seem to run concurrently. When you make an asynchronous call, it doesn’t wait for the result immediately. Instead, it moves on to execute other parts of your program and returns to the original task once it’s complete. It’s like the coffee shop employing multiple baristas so that customers can order and collect their drinks in parallel.

Why should PHP developers care about asynchronous programming?

Asynchronous programming offers significant advantages for PHP applications that need to handle multiple I/O-bound tasks, such as API requests, file uploads, or database queries. Here are some scenarios where adopting asynchronous methods in PHP can supercharge your application:

improving web scraping efficiency: If your PHP app needs to scrape data from multiple sources, waiting for each HTTP request to complete before starting the next can slow things down. Asynchronous processing lets your script send multiple HTTP requests simultaneously and handle the responses as they arrive, speeding up the scraping process dramatically.
handling multiple API requests: Applications that depend on several external APIs, like aggregating social media feeds, can benefit from non-blocking requests. Instead of waiting for each API call to return, PHP can make all the requests asynchronously, improving the overall response time.
optimizing real-time applications: Think about online gaming, real-time chats, or collaborative tools where latency can ruin the user experience. Asynchronous programming allows PHP developers to handle multiple events and user inputs at once, providing a much smoother and more responsive user experience.
background task processing: If you have tasks such as sending email confirmations, processing images, or performing background data analysis, asynchronous programming is ideal. You don’t want these background operations to slow down the main script or web request.

Exploring async libraries and tools in PHP

Historically, PHP hasn’t been the go-to language for asynchronous programming. The language’s synchronous nature made it more suited for traditional, blocking operations. However, the PHP community has developed several libraries and tools to support asynchronous programming. Let’s dive into some of the most notable ones:

1. Swoole

Swoole is a high-performance coroutine-based PHP extension that brings asynchronous I/O and concurrent programming capabilities to PHP. It allows developers to build event-driven, real-time applications using native PHP code.

Key features:

Coroutines: Enable multiple tasks to run concurrently, reducing the complexity of managing asynchronous code.
Asynchronous I/O: Supports non-blocking HTTP clients, database queries, file operations, and more.
WebSockets: Ideal for building real-time applications like live chats and dashboards.

Example: Here’s a simple example of creating an asynchronous HTTP server using Swoole:

$server = new Swoole\Http\Server("127.0.0.1", 9501);

$server->on("request", function ($request, $response) {
    $response->end("Hello, Swoole!");
});

$server->start();

With Swoole, you can build efficient servers that can handle multiple requests simultaneously without blocking.

2. ReactPHP

ReactPHP is another popular asynchronous PHP library. It’s event-driven and non-blocking, making it well-suited for I/O-intensive applications. ReactPHP can help create long-running applications like chat servers, API integrations, or even real-time analytics tools.

Key features:

Non-blocking I/O: ReactPHP allows PHP scripts to handle many I/O operations concurrently, like HTTP requests and WebSocket connections.
Event loop: Uses an event-driven model that works similarly to Node.js, enabling responsive and real-time applications.

Example: Here’s how to create an asynchronous HTTP server using ReactPHP:

use React\Http\Server;
use Psr\Http\Message\ServerRequestInterface;
use React\Http\Message\Response;
use React\EventLoop\Factory;

require 'vendor/autoload.php';

$loop = Factory::create();
$server = new Server(function (ServerRequestInterface $request) {
    return new Response(200, ['Content-Type' => 'text/plain'], "Hello, ReactPHP!");
});

$socket = new React\Socket\Server('127.0.0.1:8080', $loop);
$server->listen($socket);
$loop->run();

With ReactPHP, the event-driven nature of your code allows it to handle requests concurrently.

3. Amp

Amp is another promising library that introduces asynchronous programming using coroutines. It’s lightweight and designed to make writing asynchronous code in PHP simple and intuitive. Amp supports HTTP servers, MySQL queries, and more.

Key features:

Promise-based concurrency: Offers an easy way to work with async tasks.
Coroutines: Use generators to write non-blocking code.
Rich ecosystem: Provides multiple packages, including for HTTP servers, database queries, and WebSocket servers.

Example: An example of using Amp for making asynchronous HTTP requests:

use Amp\Loop;
use Amp\Http\Client\Request;
use Amp\Http\Client\HttpClientBuilder;

require 'vendor/autoload.php';

Loop::run(function () {
    $client = HttpClientBuilder::buildDefault();
    $request = new Request("http://example.com");

    $response = yield $client->request($request);

    echo yield $response->getBody()->buffer();
});

Amp’s promise-based approach and support for coroutines make it a great choice for PHP developers looking to add async capabilities to their applications.

Comparison of async libraries

Here’s a brief comparison of the three prominent libraries:

Feature	Swoole	ReactPHP	Amp
Programming Model	Coroutines, async I/O	Event-driven, non-blocking	Promise-based, coroutines
Real-time Apps	Great for real-time	Well-suited	Suitable for real-time
Ecosystem Support	WebSockets, HTTP	HTTP, WebSocket, TCP	HTTP, MySQL, Redis
Learning Curve	Medium	Low to medium	Medium

When to choose asynchronous PHP

While asynchronous programming offers considerable advantages, it’s not always the best solution for every PHP project. Here are some scenarios where asynchronous PHP shines:

API-heavy applications: When your application needs to consume multiple APIs simultaneously, asynchronous PHP reduces the total wait time.
real-time applications: Chat applications, gaming servers, or live analytics dashboards benefit significantly from non-blocking PHP code.
file processing: Large file uploads, image resizing, or PDF processing can be handled in the background while the rest of the application remains responsive.

Conversely, for simpler websites or applications that are mainly read-heavy (like blogs or news websites), the complexity of async PHP may not offer enough benefits to justify the additional learning curve.

Conclusion

Asynchronous programming might seem daunting to PHP developers initially, but with tools like Swoole, ReactPHP, and Amp, PHP’s asynchronous capabilities are more accessible than ever. Embracing async methods in your PHP projects can enhance performance, responsiveness, and scalability, making your applications more competitive in today’s fast-paced digital world.

By leveraging these techniques, PHP developers can expand their horizons and dive into new realms of concurrent programming, creating dynamic and highly performant applications. So, why not try out asynchronous PHP in your next project?

Sources:

Framework overload: when convenience dulls innovation in software development

2024-10-24T00:00:00+00:00

Web and software developers are more empowered today than ever. But there’s a growing problem: the increasing reliance on frameworks, libraries, and pre-packaged solutions. Frameworks like Laravel, React, Angular, and Django have become household names, not just for their ability to streamline development but also for their role in reshaping how developers approach problem-solving. While these frameworks undoubtedly improve efficiency and scalability, their dominance is also inadvertently curbing creativity and encouraging cookie-cutter solutions.

The allure of frameworks: convenience over comprehension

Let’s face it, frameworks are alluring. They offer pre-built structures, built-in security measures, and optimized performance. Why reinvent the wheel when there’s a highly polished, thoroughly tested wheel already available? This mindset is where the problem begins. For many developers, understanding how a framework functions under the hood or grasping the nuances of core languages like PHP, Python, or JavaScript has taken a backseat. Instead, the race is on to master the framework, not necessarily the foundational technology.

For example, when we work with Laravel for PHP, it abstracts much of the nitty-gritty of database management using its Eloquent ORM. While this is fantastic for developers looking to build applications quickly, it can lead to a situation where developers aren’t familiar with basic SQL concepts, such as joins, indexing, or optimization strategies. The abstraction, while convenient, becomes a crutch, limiting a deeper understanding of the language and the database systems in play.

A popular anecdote among experienced programmers is encountering job candidates who excel at following documentation to the letter but falter when presented with unique or unconventional problems. In such cases, over-reliance on frameworks has effectively dulled their problem-solving abilities.

The trap of tool-centric learning

When you focus on mastering frameworks, you end up chasing trends. In recent years, there’s been a rush to learn new tools rather than mastering the underlying principles of coding and problem-solving. This approach runs contrary to the spirit of innovation. A competent craftsman isn’t defined by their tools but by their skill, creativity, and understanding of materials.

Consider the rise of front-end frameworks like React. While these frameworks have revolutionized the web, many developers jump straight into JSX and component states without having a firm grasp of vanilla JavaScript. They are fluent in implementing lifecycle methods and hooks but are left scrambling when faced with simple tasks outside the React ecosystem.

The same applies to mobile development, where the advent of Flutter and React Native has driven many developers away from native mobile development. While hybrid frameworks are an excellent choice for cross-platform deployment, they can lead developers to a shallow understanding of how the underlying operating system works.

Missing the forest for the trees: losing sight of the bigger picture

Let’s take the gaming industry as an example. With the prevalence of powerful game engines like Unity and Unreal, developers can create complex games relatively quickly. These engines offer incredible capabilities for rendering, physics, and AI. However, this convenience often leads to an over-reliance on built-in features rather than understanding core graphics programming or physics simulation.

In web development, the complexity added by juggling multiple libraries can lead developers to fall into the trap of “over-engineering.” When a relatively simple problem is approached with overly complicated solutions, developers often lose sight of the core purpose and creativity behind their work. The technical focus becomes how to optimize their React code or fine-tune a Redux store, rather than solving the user’s actual problem.

This is where the phrase “missing the forest for the trees” applies. Developers sometimes become so engrossed in the tools at their disposal that they fail to step back and evaluate whether these tools are necessary or the best fit for the job.

Frameworks as enablers, not crutches: a path forward

The solution isn’t to stop using frameworks altogether—they’re valuable and time-saving. However, a balanced approach is key. Here are some recommendations to prevent frameworks from dulling innovation:

Learn the basics thoroughly: Commit to understanding core concepts in programming languages before jumping to frameworks. Know what happens under the hood. For example, a deep dive into PHP’s handling of HTTP requests or JavaScript’s prototype-based inheritance can be eye-opening.
Don’t skip problem-solving skills: The best way to learn coding is not by following tutorials blindly, but by working on projects that challenge you to solve unique problems. Instead of relying solely on predefined solutions, challenge yourself to come up with your own, even if they are less polished or more cumbersome initially.
Use frameworks as scaffolding, not blueprints: Think of frameworks as scaffolding—they provide the structure and support needed to erect a building. However, the design of the building itself should come from the architect (you). It’s essential to keep the focus on the problem and not allow frameworks to dictate every aspect of your solution.
Experiment beyond the framework: Spend time creating projects that are not tied to any framework. For instance, try creating a simple REST API without Laravel or Spring. This forces you to engage with the raw HTTP protocol and routing mechanisms, broadening your understanding of how these frameworks streamline tasks.
Embrace minimalism: Before you install yet another package, ask yourself if you genuinely need it. Consider the words of Steve Jobs: “Innovation is saying no to a thousand things.” Applying this minimalist philosophy to your tech stack can keep your projects simple and impactful.

Conclusion: towards a balanced approach

In the end, frameworks are here to stay, and rightly so—they save time, reduce complexity, and make our lives easier. But it’s essential to remember that they are not the goal; they’re merely a means to an end. Developers should strive to build a solid understanding of fundamental concepts, focusing on solving problems creatively rather than fitting them into the constraints of a particular framework.

We must remember that the tools we use should enhance our creativity and not limit it. By approaching frameworks with this balanced mindset, we can continue to innovate, solve complex problems, and create truly remarkable work.

Guarding the digital frontier: top open source security projects of 2024

2024-10-20T00:00:00+00:00

As digital threats evolve at an unprecedented pace, robust cybersecurity measures have become more critical than ever. Open source security projects stand at the forefront of this battle, offering powerful tools that empower organizations to protect their data and infrastructure. These projects promote transparency, encourage collaboration, and often provide cost-effective alternatives to proprietary software. In 2024, the open source community continues to innovate, with numerous security projects making significant strides in safeguarding the digital frontier. This article will explore the leading open source security projects you should know about this year.

The importance of open source in cybersecurity

Open source security solutions have become indispensable in today’s cybersecurity landscape for several reasons:

1. Transparency

The open source model allows anyone to inspect the code, making it easier to identify vulnerabilities. This transparency builds trust among users, as they can verify security measures themselves.

2. Community collaboration

The collaborative nature of open source allows developers and security experts worldwide to contribute, leading to more robust and constantly evolving solutions.

3. Cost-effectiveness

Many open source tools are free, making them accessible to organizations of all sizes. This is particularly beneficial for small businesses and startups with limited cybersecurity budgets.

4. Flexibility and customization

Organizations can modify open source projects to suit their specific security needs, providing a tailored solution that proprietary software may not offer.

5. Rapid innovation

With contributions from a diverse pool of developers, open source security projects often adopt the latest security technologies and trends faster than proprietary alternatives.

Top open source security projects of 2024

Here are some of the most impactful open source security projects in 2024, helping organizations bolster their defenses against modern cyber threats:

1. OWASP ZAP (Zed Attack Proxy)

Overview: OWASP ZAP is an advanced tool for discovering security vulnerabilities in web applications. It is user-friendly and suitable for both beginners and experienced professionals.
Why It Matters: As web applications grow more complex, OWASP ZAP remains a popular choice for developers and security teams to identify and remediate vulnerabilities effectively.

2. Metasploit Framework

Overview: Metasploit is a powerful penetration testing framework used to find and exploit system vulnerabilities.
Why It Matters: Metasploit is widely relied upon by ethical hackers for simulating real-world attacks, helping organizations strengthen their cybersecurity defenses.

3. Suricata

Overview: Suricata is an open source network threat detection engine that supports intrusion detection, prevention, and security monitoring.
Why It Matters: Suricata’s ability to analyze network traffic in real-time provides essential protection against evolving cyber threats.

4. ClamAV

Overview: ClamAV is an antivirus engine that helps detect viruses, malware, and malicious threats.
Why It Matters: ClamAV is commonly used in environments that require consistent malware scanning, including mail servers and web applications.

5. WireGuard

Overview: WireGuard is a modern VPN protocol designed for speed and simplicity, offering secure communication with minimal complexity.
Why It Matters: As remote work and online privacy concerns rise, WireGuard is gaining traction for its efficiency and strong encryption standards.

6. OpenVAS

Overview: OpenVAS (Open Vulnerability Assessment System) is a comprehensive vulnerability scanning tool that helps identify weaknesses in networks and systems.
Why It Matters: Regular vulnerability assessments with tools like OpenVAS are essential for staying ahead of emerging cyber threats.

7. TheHive

Overview: TheHive is an incident response platform that enables teams to manage security incidents collaboratively.
Why It Matters: In a landscape where cyber incidents can escalate quickly, TheHive provides the tools necessary for effective incident management.

8. OSSEC

Overview: OSSEC is a host-based intrusion detection system (HIDS) that monitors and analyzes logs, identifying anomalies and potential threats.
Why It Matters: OSSEC plays a key role in maintaining system security and compliance by offering insights into system behavior.

9. Kali Linux

Overview: Kali Linux is a popular Debian-based distribution designed for penetration testing and ethical hacking, featuring pre-installed security tools.
Why It Matters: Kali Linux remains a favorite for security practitioners conducting network scans, exploit development, and web application testing.

10. Lynis

Overview: Lynis is a security auditing tool for Unix-based systems that performs in-depth scans and offers suggestions for improving security.
Why It Matters: Lynis helps organizations assess their security posture and implement best practices to mitigate potential risks.

How to Get Involved in Open Source Security Projects

Contributing to open source security projects can boost your skills and positively impact the cybersecurity community. Here’s how to get started:

1. Choose a project

Find a project that aligns with your interests and skills. Many projects list beginner-friendly tasks under labels like “good first issue” or “help wanted.”

2. Set up your development environment

Familiarize yourself with the project’s tools and technologies, such as Git and specific programming languages.

3. Follow contribution guidelines

Each project has contribution guidelines. Understanding how to submit issues, pull requests, and code reviews is essential.

4. Start small

Begin with manageable tasks, such as fixing documentation or small bugs, to build confidence and get familiar with the codebase.

5. Engage with the community

Participating in community discussions can help you gain valuable insights and establish relationships with fellow contributors.

6. Be open to feedback

Contributions often come with feedback. Use it as an opportunity to learn and improve your skills.

Conclusion

Open source security projects are transforming how organizations defend against cyber threats. Leading tools like OWASP ZAP, Metasploit, and Suricata are paving the way for innovative, transparent, and collaborative cybersecurity solutions. As we navigate 2024’s complex cybersecurity landscape, contributing to these projects can help protect digital assets while enhancing your professional growth. Embrace the power of collaboration in the open source community, and together, let’s guard the digital frontier!

Sources:

Why forced updates in Snaps and Flatpaks are breaking Linux's core principles

2024-10-19T00:00:00+00:00

Linux has long been celebrated for its commitment to user control and freedom. However, the rise of modern packaging systems like Snaps and Flatpaks, with their mandatory updates, poses a significant challenge to these core principles. This article delves into the implications of forced updates, highlighting how they disrupt user autonomy, compromise system stability, and contribute to the growing concern of vendor lock-in.

User control vs. convenience: The forced updates issue

Traditional Linux packaging systems, such as DEBs and RPMs, prioritize user control, enabling individuals to decide when and how to update their software. In contrast, Snaps and Flatpaks cater to convenience, especially for less technical users, by offering automated updates and dependency management. Yet, this shift raises critical questions about user autonomy.

For many, forced updates can disrupt workflows, especially in environments where system stability is essential. The need to maintain consistent software versions becomes paramount, as unexpected updates may introduce bugs or break compatibility with existing applications.

The downsides of forced updates: breaking the ‘Linux Philosophy’

Linux distributions thrive on the “do one thing and do it well” philosophy. However, Snaps and Flatpaks bundle dependencies together, which, while reducing “dependency hell,” results in larger package sizes and unintended updates. The backlash against Snap’s background updates is telling; users often find themselves unprepared for changes that can occur without prior notice. This lack of transparency runs counter to the traditional Linux approach, where users are empowered to control their systems.

In mission-critical environments—such as servers or development setups—unplanned updates can be particularly detrimental. Users may find themselves scrambling to revert to previous versions after a sudden break in functionality, illustrating the tension between convenience and control.

Security vs. Stability: Striking the balance

Proponents of forced updates argue that they enhance security by ensuring vulnerabilities are patched promptly. However, this perspective overlooks the stability concerns of users who prefer to test updates before implementation. For many, particularly those managing critical Linux systems, the desire to maintain operational integrity outweighs the perceived security benefits of automatic updates.

A well-balanced approach would allow users to review updates, implement testing phases, and ensure compatibility before deploying changes. This flexibility fosters a more resilient system, prioritizing both security and user autonomy.

Bloat and performance issues

Another significant critique of Snaps and Flatpaks is their tendency to create bloat. By packaging all dependencies with each application, these systems lead to larger file sizes and decreased performance compared to traditional package managers. The forced update mechanism compounds this issue, as it continually downloads newer versions, consuming bandwidth and storage space, even when the current version meets the user’s needs.

This inefficiency can be particularly frustrating for those who value a lean and responsive system. As the Linux community increasingly embraces minimalism, the bloat associated with modern packaging systems stands in stark contrast to these ideals.

Vendor lock-in concerns

While Snaps are closely associated with Canonical, the parent company of Ubuntu, Flatpaks are tied to Red Hat and GNOME. This centralization raises alarms within the Linux community, which has historically championed decentralization and user choice. Forced updates can create a sense of dependency on specific companies’ methodologies, eroding the freedom that Linux users have come to expect.

App store mentality: convenience vs. freedom

Snaps and Flatpaks aim to streamline the software distribution process, aligning Linux with the app store models familiar to users of macOS and Windows. While this approach simplifies access to applications, it also detracts from Linux’s foundational principles of user control and customization. The reality of forced updates can feel like a compromise of freedom in exchange for the illusion of convenience.

Potential solutions and middle ground

To address these concerns, developers could consider implementing more granular controls over Snap and Flatpak updates. Options for users to opt-in or opt-out of forced updates could restore some autonomy, allowing for a more tailored experience. Additionally, offering Long-Term Support (LTS) versions of packages could provide users with stability while ensuring that security updates are still accessible when needed.

Alternative package management systems

Other packaging solutions, such as AppImages, provide an appealing alternative by allowing for portable applications without enforced updates. Furthermore, traditional package management systems (like APT or RPM) continue to hold value, especially in scenarios where stability and control are paramount. By examining the pros and cons of these systems compared to newer methodologies, users can make informed decisions about the best approach for their needs.

Conclusion

The debate surrounding forced updates in Snaps and Flatpaks resonates deeply with the Linux community. While these modern packaging systems aim to enhance convenience, they risk undermining the core principles of user control, stability, and decentralization that have defined Linux for decades. As discussions about the future of Linux packaging continue, striking a balance between attracting new users and preserving the values that long-time enthusiasts cherish will be crucial. In a landscape where user autonomy is increasingly challenged, it’s essential to advocate for solutions that empower users rather than constrain them.

Where to get your icons for web development: a comprehensive guide

2024-10-18T00:00:00+00:00

Icons are an essential part of modern web development, providing visual cues that enhance user experience and making interfaces more intuitive and appealing. However, choosing the right set of icons involves considering both aesthetic appeal and licensing terms. In this article, we’ll explore three primary sources for web development icons: truly open-source icons, “with attribution” options, and proprietary ones. Let’s dive into these categories to help you make an informed decision.

1. Truly open-source icons

If you’re looking for full flexibility—where you can modify, distribute, and use icons without worrying about commercial restrictions—open-source icons are your best bet. These icon sets often come with permissive licenses, allowing for broad usage in both personal and commercial projects.

Adwaita (GNOME Icons)

The Adwaita icon theme, used in the GNOME desktop environment, is an excellent option for developers who prioritize open-source freedom. The icon set follows the Creative Commons Attribution-ShareAlike 3.0 License (CC-BY-SA 3.0), which allows you to:

Use the icons freely in any project, including commercial ones.
Modify the icons to suit your design needs.
Share them, provided that you attribute the original creators and maintain the same license for any derivative work.

Where to Download: You can download GNOME’s Adwaita icons from their GitLab repository.

Material Design Icons

Material Design Icons are inspired by Google’s Material Design guidelines and are widely used in web and app development. This set is licensed under the Apache License 2.0, a permissive open-source license. You can use, modify, and distribute the icons freely, but you must include a copy of the license in any redistribution.

Where to Download: You can find the full set of Material Design Icons on their GitHub repository.

Font Awesome (Open-Source Icons)

Font Awesome offers an extensive library of icons with both free and paid options. The free icons are licensed under Creative Commons Attribution 4.0 and can be used for both personal and commercial projects, provided attribution is given.

Where to Download: Visit Font Awesome’s website to download the free icons.

2. “With Attribution” icon resources

If you’re open to giving credit to the original creators, “with attribution” icon sets offer a middle ground between fully open-source and proprietary. These platforms typically provide free icons but require proper attribution to be used legally. Here’s a look at some popular options in this category.

Flaticon

Flaticon is one of the largest databases of free icons, offering millions of icons in various formats. You can use the free version of the icons as long as you provide attribution. The platform also offers a premium option, which removes the attribution requirement.

License: Free icons require attribution under a custom license.
Use Case: Perfect for quick access to vast icon libraries, but you’ll need to credit Flaticon if you use their free icons in a commercial or personal project.

Where to Download: Visit Flaticon to explore their collection.

Freepik

Freepik provides not only icons but also a wide range of design assets, including vectors, illustrations, and photos. Freepik’s free resources come with the same attribution requirement, while their premium subscription offers attribution-free usage.

License: Free resources are available under the Freepik License, which requires attribution. Premium resources are licensed under more relaxed terms.
Use Case: A great resource for comprehensive design needs beyond icons, but attribution is a must for free content.

Where to Download: You can access Freepik’s collection at Freepik.com.

Icons8

Icons8 offers a collection of free icons for use in web development. Like Flaticon and Freepik, their free icons require attribution, though premium users can avoid this requirement.

License: Free icons need attribution, but premium licenses remove this requirement.
Use Case: Icons8 also provides tools for customizing and editing icons, making it useful for developers looking for more flexibility.

Where to Download: Explore their library on Icons8’s website.

3. Proprietary icon resources

For developers who need high-quality, professional icons and are willing to invest in premium licenses, proprietary options offer the most polished and diverse collections. These icons come with specific licensing terms that restrict redistribution but offer more freedom in commercial use without needing attribution.

Adobe Stock Icons

Adobe Stock offers a large variety of professionally designed icons as part of its stock asset library. These icons are royalty-free but come with strict licensing terms that allow use only in specific commercial projects, without redistribution rights.

License: Icons purchased from Adobe Stock are typically licensed under Adobe’s royalty-free stock licensing, allowing use in personal and commercial projects but prohibiting redistribution or modification beyond project needs.
Use Case: Best suited for high-end commercial projects that require polished, unique iconography.

Where to Download: You can browse Adobe Stock icons at Adobe Stock’s official site.

Envato Elements

Envato Elements is another popular marketplace for premium design resources, including icons. It offers a subscription-based model where you get access to all design assets, including icons, under a commercial license.

License: The assets on Envato Elements are available under their own commercial license, which allows unlimited use in projects but does not allow redistribution of the icons.
Use Case: Suitable for businesses and developers looking for exclusive, high-quality design assets for client work or branding purposes.

Where to Download: Visit Envato Elements for more information.

Conclusion

Choosing the right icon set depends on your project requirements and licensing preferences:

Truly Open-Source icons like Adwaita and Material Design Icons offer the most flexibility for open-source projects, allowing full use and modification with permissive licenses.
“With Attribution” platforms like Flaticon and Freepik offer vast collections of free icons but require giving credit to the creators.
Proprietary Options like Adobe Stock and Envato Elements are ideal for high-quality, professional-grade icons, but come with stricter usage rights and licensing fees.

By carefully selecting the right source for your icons, you can enhance your web development projects while adhering to legal and ethical guidelines.

Sources

Your go-to guide to open source forums: where developers collaborate and innovate

2024-10-17T00:00:00+00:00

In the ever-evolving world of technology, open source software stands out as a beacon of collaboration, creativity, and community spirit. As developers dive into the realm of open source, forums play a crucial role in facilitating discussions, sharing knowledge, and fostering innovation. This guide will explore the significance of open source forums, the best platforms to engage with, and tips on how to make the most of these collaborative spaces.

Why Open Source Forums Matter

Open source forums serve as virtual meeting grounds where developers, contributors, and enthusiasts can come together to share ideas, troubleshoot problems, and collaborate on projects. Here are several reasons why these forums are vital to the open source ecosystem:

1. Community Support

Forums offer a platform for users to seek help and advice. Whether you’re a beginner grappling with your first code or an experienced developer facing a complex issue, forums connect you with a supportive community .

Developers often share tutorials, documentation, and resources in forums, which helps others learn and grow. This culture of sharing is fundamental to the success of open source projects .

3. Networking Opportunities

Open source forums allow you to connect with like-minded individuals, potential collaborators, and industry professionals. These connections can lead to job opportunities, mentorship, and partnerships .

4. Collaboration and Innovation

Many open source projects thrive on community contributions. Forums often serve as the starting point for collaboration, where ideas can be discussed and implemented, leading to innovative solutions .

Top Open Source Forums to Join

As you embark on your journey into the open source world, consider joining these well-established forums where developers collaborate and innovate:

1. Stack Overflow

Overview: Stack Overflow is one of the largest Q&A platforms for developers. It covers a vast range of programming languages and technologies, making it a go-to resource for troubleshooting and learning.
Why Join?: With a robust tagging system, you can find questions and answers related to specific open source projects, frameworks, and programming languages. Contributing to discussions enhances your reputation and visibility within the community .

2. Reddit

Overview: Reddit hosts a plethora of subreddits dedicated to open source software and development, such as r/opensource, r/Python, and r/programming.
Why Join?: Reddit’s informal nature encourages open discussions, idea sharing, and project promotion. Engaging in subreddit communities allows you to connect with fellow enthusiasts and learn from diverse perspectives .

3. GitHub Discussions

Overview: GitHub Discussions is a feature integrated into GitHub repositories, enabling users to ask questions, share ideas, and discuss features related to specific projects.
Why Join?: This forum provides direct access to the project maintainers and contributors. It’s an ideal space for brainstorming and providing feedback on ongoing projects .

4. OpenSource.com

Overview: OpenSource.com is a platform dedicated to sharing stories, resources, and insights about open source. It features a vibrant community of writers and contributors.
Why Join?: By engaging with articles and participating in discussions, you can stay updated on open source trends and contribute your insights, fostering a richer understanding of the ecosystem .

5. The Linux Foundation Forums

Overview: The Linux Foundation hosts forums that cater specifically to Linux users and developers. These forums are a valuable resource for discussions on Linux-based projects and initiatives.
Why Join?: Being part of the Linux Foundation community provides access to valuable resources, training, and networking opportunities with industry leaders .

6. Dev.to

Overview: Dev.to is a community of software developers who share articles, tutorials, and insights. It fosters a friendly and inclusive environment for learning and collaboration.
Why Join?: The platform encourages you to write and share your experiences, while also engaging with others through comments and discussions. This can lead to valuable feedback and new connections .

7. SourceForge Community Forums

Overview: SourceForge, one of the original platforms for hosting open source projects, offers community forums for discussing projects, features, and best practices.
Why Join?: Engaging in SourceForge forums allows you to connect with project maintainers and users, helping you learn more about specific tools and how to contribute effectively .

8. Lobsters

Overview: Lobsters is a community-driven link aggregation site focused on software development and technology, including open-source projects.
Why Join?: Its smaller, more focused community allows for high-quality discussions on technical topics, including open-source software and development tools.

9. FreeCodeCamp Forum

Overview: FreeCodeCamp is a community-driven platform offering coding tutorials, certification courses, and a large forum for learning open-source development.
Why Join?: Ideal for beginners, FreeCodeCamp allows users to discuss their learning journeys, contribute to open-source projects, and seek guidance from experienced developers.

10. Gitter

Overview: Gitter is an open-source chat platform designed for communities to discuss software development. It’s commonly used for real-time collaboration within GitHub projects.
Why Join?: Gitter offers focused discussions around specific open-source projects, helping users to engage directly with maintainers and contributors in a chat-like setting.

11. Geeksmint Forums

Overview: Geeksmint is a platform focused on free and open-source software (FOSS) news and tutorials, and it offers community forums for in-depth discussions.
Why Join?: It’s an excellent resource for discovering lesser-known open-source tools and getting advice from a diverse and global FOSS community.

12. Discourse Meta

Overview: Discourse Meta is a community discussion platform built with Discourse. It’s commonly used by open-source developers to discuss feature requests, bug reports, and project enhancements.
Why Join?: Many open-source projects use Discourse as their primary discussion forum, so joining allows you to engage directly with various project communities.

13. Hacker News

Overview: Hacker News is a social news site focused on technology and startups. While not strictly a forum, it has a strong developer community and is known for insightful discussions about open-source trends and tools.
Why Join?: Participating in discussions on Hacker News can give you access to cutting-edge insights and opportunities to connect with thought leaders in the tech industry.

14. OpenSUSE Forums

Overview: OpenSUSE, a popular Linux distribution, offers a forum for users and developers to discuss its ecosystem and open-source software in general.
Why Join?: This community is ideal for Linux enthusiasts looking to engage with a dedicated group of open-source software users and developers.

15. Mozilla Discourse

Overview: Mozilla, the organization behind Firefox, provides a Discourse forum for users and developers to discuss its various open-source projects, from browsers to privacy tools.
Why Join?: It’s an opportunity to directly influence one of the most prominent open-source organizations and engage with Mozilla’s developer community.

16. Apache Software Foundation Mailing Lists

Overview: The Apache Software Foundation offers mailing lists that function as forums for open-source projects under its umbrella, including Hadoop, Spark, and HTTP Server.
Why Join?: Mailing lists are an excellent way to stay up-to-date on project developments, seek help, and contribute to Apache’s many open-source initiatives.

How to Make the Most of Open Source Forums

Participating in open source forums can be immensely rewarding, but it requires some effort to maximize your experience. Here are some tips to help you engage effectively:

1. Be Respectful and Constructive

When engaging in discussions, always maintain a respectful tone. Constructive criticism and positive feedback foster a healthy community atmosphere.

2. Ask Thoughtful Questions

If you’re seeking help, ensure your questions are clear and specific. Include relevant details, such as error messages and the context of your problem. This makes it easier for others to assist you.

3. Contribute to Discussions

Don’t hesitate to share your knowledge or experiences. Contributing to discussions enhances your reputation and helps others in the community.

4. Follow Forum Etiquette

Familiarize yourself with each forum’s rules and guidelines. This ensures your contributions are well-received and align with community standards.

5. Stay Active and Engaged

Regularly participate in discussions, comment on articles, and follow up on questions you’ve asked. Staying active increases your visibility and fosters connections.

Conclusion

Open source forums are invaluable resources for developers at all skill levels. They provide a platform for collaboration, learning, and innovation that shapes the future of software development. By engaging with these communities, you’ll not only enhance your own skills but also contribute to a vibrant ecosystem that thrives on shared knowledge and collaboration.

So, whether you’re troubleshooting an issue, seeking advice, or sharing your insights, dive into the world of open source forums. Join the conversation, connect with fellow developers, and become an integral part of the open source movement!

Sources:

Stack Overflow: https://stackoverflow.com
OpenSource.com: https://opensource.com
The Linux Foundation: https://linuxfoundation.org
Reddit: https://www.reddit.com
GitHub Discussions: https://github.com
Dev.to: https://dev.to
SourceForge: https://sourceforge.net
Lobsters: https://lobste.rs
FreeCodeCamp Forum: https://forum.freecodecamp.org
Gitter: https://gitter.im
Geeksmint Forums: https://www.geeksmint.com/
Discourse Meta: https://meta.discourse.org
Hacker News: https://news.ycombinator.com
OpenSUSE Forums: https://forums.opensuse.org
Mozilla Discourse: https://discourse.mozilla.org
Apache Software Foundation Mailing Lists: https://lists.apache.org

Open source in education: empowering students and teachers

2024-10-16T00:00:00+00:00

In the digital age, technology has become an indispensable tool in education. Open-source software, with its free, accessible, and collaborative nature, offers a powerful means to revolutionize teaching and learning. This article delves into the ways in which open-source solutions can empower both students and teachers, fostering a more equitable, innovative, and personalized educational experience.

Empowering Students: Fostering Creativity and Critical Thinking

Open-source software provides students with a unique opportunity to engage with technology on a deeper level. By interacting with open-source projects, students can:

Develop Problem-Solving Skills: Open-source software often requires users to troubleshoot issues and find solutions. This process helps students develop essential problem-solving skills that are invaluable in both academic and professional settings.
Cultivate Critical Thinking: The open-source model encourages students to question and analyze technology. By understanding how software works, students can develop a critical perspective on its use and limitations.
Nurture Creativity: Open-source platforms offer a space for students to experiment and innovate. They can contribute to existing projects, create their own, or customize existing software to suit their needs. This fosters creativity and encourages students to think outside the box.
Learn Collaborative Skills: Open-source development often involves collaboration with others from around the world. This provides students with opportunities to learn how to work effectively in teams, communicate effectively, and respect diverse perspectives.

Empowering Teachers: Enhancing Teaching and Learning

Open-source software can also significantly enhance the teaching experience. By leveraging open-source tools, teachers can:

Create Personalized Learning Experiences: Open-source software can be customized to meet the specific needs of individual students. Teachers can create differentiated assignments, provide targeted support, and offer personalized learning paths.
Access High-Quality Educational Resources: A vast array of open-source educational resources, including textbooks, lesson plans, and multimedia content, are available online. This can help teachers save time and resources while ensuring that students have access to high-quality materials.
Foster a Culture of Innovation: Open-source software encourages experimentation and innovation. Teachers can use open-source tools to develop new teaching methods, create engaging activities, and explore innovative approaches to learning.
Reduce Costs: Open-source software is often free or low-cost, making it a more affordable option for schools and educators. This can help to reduce the financial burden on schools and ensure that all students have access to necessary technology.

Case Studies: Successful Open-Source Initiatives in Education

Several successful open-source initiatives have demonstrated the transformative potential of open-source software in education. Some notable examples include:

Khan Academy: This popular online learning platform uses open-source technology to provide free, world-class education to anyone, anywhere. Khan Academy’s open-source approach has enabled it to reach millions of students around the globe and has inspired countless other educational initiatives.
Sugar Labs: Sugar Labs develops the Sugar interface, which is used in the XO laptop, a low-cost, rugged laptop designed for education. Sugar Labs’ open-source approach has made it possible to provide affordable computing devices and educational software to students in developing countries.
Mozilla Foundation: Mozilla develops open-source software, such as Firefox, that is used by millions of students and educators worldwide. Mozilla’s commitment to open-source development has helped to create a more open and accessible internet, empowering users to learn and collaborate freely.

Challenges and Opportunities

While open-source software offers numerous benefits, there are also challenges to consider. These include:

Technical Support: Ensuring adequate technical support for open-source software in educational settings can be a challenge, especially for schools with limited IT resources.
Data Privacy and Security: Protecting student data and ensuring the security of open-source software is a critical concern. Schools must implement appropriate security measures to safeguard sensitive information.
Integration with Existing Systems: Integrating open-source tools with existing school infrastructure can be time-consuming and complex. Schools may need to invest in professional services or develop custom solutions to ensure compatibility.

Despite these challenges, the potential benefits of open-source software in education far outweigh the drawbacks. By embracing open-source solutions, schools and educators can create more equitable, innovative, and personalized learning environments for students.

Conclusion

Open-source software has the power to transform education by empowering students and teachers. By providing access to free, customizable, and collaborative tools, open-source can foster creativity, critical thinking, and a lifelong love of learning. As technology continues to evolve, the role of open-source in education will only become more important. By embracing open-source, schools and educators can help to create a brighter future for students around the world.

References:

Creating magic with code: simple PHP programs for beginners

2024-10-15T00:00:00+00:00

Welcome to the enchanting world of PHP! If you’ve ever dreamed of crafting your own web applications, managing databases, or simply adding a dash of interactivity to your website, you’ve stumbled upon the right place. In this article, we’ll explore some simple yet magical PHP programs that will serve as a fantastic introduction to this versatile scripting language. So, roll up your sleeves, and let’s dive in!

What is PHP?

PHP, which stands for Hypertext Preprocessor, is a server-side scripting language designed primarily for web development. It allows you to create dynamic content that can interact with databases, handle forms, and manage sessions, making it an essential tool in a web developer’s toolkit.

But wait—what makes PHP so appealing to beginners? Here are a few reasons:

Easy to Learn: With its straightforward syntax, even novices can grasp the basics quickly.
Versatile: PHP can be used for everything from simple websites to complex web applications.
Open Source: Being free and widely supported means you’ll find plenty of resources and community help along the way.

Getting Started with PHP

Before we jump into some cool programs, let’s ensure you have the essentials set up. You’ll need a local server environment to run your PHP code. XAMPP and MAMP are excellent options that allow you to create a local web server on your computer.

Once installed, place your PHP files in the htdocs directory (XAMPP) or the htdocs equivalent (MAMP) to run them. Now, let’s explore some beginner-friendly programs!

1. Hello, World!

Let’s start with the classic Hello, World! program. It’s a rite of passage for any programmer.

<?php
echo "Hello, World!";
?>

Explanation

<?php ... ?>: This tag tells the server that the code inside is PHP.
echo: This function outputs the text to the browser.

What You Learn

This simple program introduces you to the PHP syntax and the echo function.

2. Basic Calculator

Next up, we’ll create a basic calculator that can perform addition, subtraction, multiplication, and division.

<?php
function calculator($num1, $num2, $operation) {
    switch ($operation) {
        case 'add':
            return $num1 + $num2;
        case 'subtract':
            return $num1 - $num2;
        case 'multiply':
            return $num1 * $num2;
        case 'divide':
            return $num2 != 0 ? $num1 / $num2 : 'Cannot divide by zero';
        default:
            return 'Invalid operation';
    }
}

// Example Usage
echo calculator(10, 5, 'add'); // Outputs: 15
?>

Explanation

Function: The calculator function takes two numbers and an operation as parameters and returns the result based on the specified operation.
Switch Case: This structure helps to execute different code blocks based on the value of the $operation variable.

What You Learn

You’ll gain insights into functions, control structures, and how to handle basic arithmetic operations in PHP.

3. Form Handling

Now, let’s create a simple web form that collects user input. We’ll build a contact form that captures the user’s name and email.

<!DOCTYPE html>
<html>
<head>
    <title>Contact Form</title>
</head>
<body>

<form method="post" action="">
    Name: <input type="text" name="name" required><br>
    Email: <input type="email" name="email" required><br>
    <input type="submit" value="Submit">
</form>

<?php
if ($_SERVER["REQUEST_METHOD"] == "POST") {
    $name = htmlspecialchars($_POST['name']);
    $email = htmlspecialchars($_POST['email']);
    echo "Thank you, $name! Your email, $email, has been received.";
}
?>

</body>
</html>

Explanation

Form: The form uses the POST method to send data to the same script for processing.
HTML Special Characters: The htmlspecialchars function prevents XSS attacks by escaping special characters.

What You Learn

This program introduces form handling, user input validation, and secure output in PHP.

4. Array Manipulation

Let’s dive into arrays—a fundamental aspect of PHP programming. We’ll create a simple program that sorts an array of numbers.

<?php
$numbers = [5, 2, 9, 1, 5, 6];
sort($numbers);
echo "Sorted numbers: ";
foreach ($numbers as $number) {
    echo $number . " ";
}
?>

Explanation

Array: $numbers is an array that contains a list of integers.
Sort Function: The sort function organizes the array in ascending order.
Foreach Loop: This loop iterates over each element in the array.

What You Learn

You’ll discover how to work with arrays, sort them, and loop through their elements.

5. Session Management

Sessions allow you to maintain user data across different pages. Let’s create a simple login system using sessions.

<?php
session_start();

if ($_SERVER["REQUEST_METHOD"] == "POST") {
    $_SESSION['username'] = htmlspecialchars($_POST['username']);
    echo "Welcome, " . $_SESSION['username'] . "!";
}
?>

<!DOCTYPE html>
<html>
<head>
    <title>Login</title>
</head>
<body>

<form method="post" action="">
    Username: <input type="text" name="username" required><br>
    <input type="submit" value="Login">
</form>

</body>
</html>

Explanation

Session Start: session_start() initializes a new session or resumes the existing one.
Storing Data: User input is stored in the session, allowing you to access it on other pages.

What You Learn

You’ll grasp the concept of sessions and how they help maintain state in web applications.

Conclusion

Congratulations! You’ve just scratched the surface of what PHP can do. From simple programs like a calculator to managing user sessions, these examples should give you a solid foundation in PHP.

Remember, coding is like magic; the more you practice, the more powerful your spells (or programs) will become. Keep experimenting, building, and pushing the boundaries of your creativity. Who knows? One day, you might just conjure up the next big web application!

Now, go forth and create some magic with code!

Python tutorial for beginners: learn the basics

2024-10-14T00:00:00+00:00

Welcome to the wonderful world of Python programming! If you’re new to coding, buckle up because Python is one of the easiest, yet most powerful, languages out there. Whether you’re looking to automate tedious tasks, build web apps, or dive into data science, Python is your gateway to coding success.

In this beginner’s guide, we’ll walk you through the essential building blocks of Python, making sure you’re ready to tackle more advanced projects in no time. Let’s get started!

Why Python?

Before we jump into the nuts and bolts, let’s talk about why Python is an excellent choice for beginners.

Imagine coding as learning to drive. Python is like the automatic transmission of programming languages — smooth, easy, and efficient. You don’t need to worry about complicated syntax rules or puzzling error messages. Its straightforward design allows you to focus on learning how to think like a programmer, not battling the language itself.

Installing Python: Your First Step

First things first — you need to install Python on your machine. Go to python.org and download the latest version (Python 3 is recommended). Installation is a breeze, and you’ll have Python running on your machine within minutes. Once installed, open your terminal or command prompt and type python --version to ensure everything’s set up.

Your First Python Program

Every programmer’s journey begins with the iconic “Hello, World!” program. It’s the software equivalent of a warm hug. Open your favorite code editor (you can use a simple text editor, or for a more professional feel, try IDEs like PyCharm or VS Code), and type:

print("Hello, World!")

Run your script, and you should see that familiar greeting pop up on your screen. Congratulations! You just wrote your first Python program. Now, let’s build on that momentum.

Variables: Python’s Building Blocks

Variables are like little containers that hold data. In Python, creating a variable is as simple as assigning a value to a name:

age = 25
name = "John"
is_student = True

No need to declare the type of the variable (whether it’s a number, a string, or a boolean) — Python figures it out for you. It’s as if Python is a mind reader, but with code.

You can manipulate these variables however you like:

print(name, "is", age, "years old.")

Python allows you to easily print out these values and even combine them in a readable format. This is one of the language’s best features — simplicity without sacrificing power.

Data Types in Python

Python uses several basic data types, each suited for a different kind of task. Here are the most common ones you’ll encounter:

Integers: Whole numbers (e.g., 10, 42, -3)
Floats: Numbers with decimal points (e.g., 3.14, 0.99)
Strings: A sequence of characters (e.g., “Hello”, “Python”)
Booleans: True or False values (e.g., True, False)

Each data type has its unique properties, and as you grow more experienced, you’ll start recognizing when and how to use them effectively.

Lists: Grouping Data Together

Lists are one of Python’s most versatile data structures. Think of a list as a collection of items (just like your shopping list) that you can easily manipulate:

fruits = ["apple", "banana", "cherry"]

You can access individual items using an index:

print(fruits[0])  # This will print "apple"

And you can even add or remove items from the list:

fruits.append("orange")  # Adds "orange" to the list
fruits.remove("banana")  # Removes "banana" from the list

Lists can grow and shrink dynamically, and they’re perfect for scenarios where you need to store multiple values.

Control Flow: If-Else Statements

Now that we’ve got data, what if we want our program to make decisions? This is where conditional statements like if and else come into play.

age = 18

if age >= 18:
    print("You’re an adult.")
else:
    print("You’re still a minor.")

Python’s syntax for conditions is as readable as it gets. No need for excessive punctuation — just a clean, human-readable format that tells you exactly what’s going on.

Loops: Repeating Actions

Repetition is a key part of programming. Loops allow you to repeat certain actions without writing the same code over and over.

For Loop

A for loop lets you iterate over a sequence (like a list or a range of numbers):

for fruit in fruits:
    print(fruit)

This will print each fruit in the fruits list, one by one.

While Loop

If you want to repeat something as long as a condition is true, use a while loop:

count = 0

while count < 5:
    print(count)
    count += 1  # This increases count by 1 each time

This loop runs until the condition (count < 5) is no longer true. It’s perfect for situations where the number of iterations isn’t fixed ahead of time.

Functions: Reusing Code

Functions are your way of bundling code into reusable blocks. Instead of writing the same code over and over, you can create a function and call it whenever needed:

def greet(name):
    print("Hello, " + name)

greet("Alice")  # This will print "Hello, Alice"
greet("Bob")    # This will print "Hello, Bob"

Functions are the secret to writing clean, efficient code. They also make your programs easier to read and maintain.

User Input: Making Your Program Interactive

Why keep all the fun to yourself? Let’s make your program interactive by allowing users to input data.

name = input("Enter your name: ")
print("Hello, " + name + "!")

This little script prompts the user for their name and responds with a personalized greeting. It’s a great way to make your programs feel dynamic and responsive.

Conclusion

Congratulations, you’ve just dipped your toes into the basics of Python programming! From writing your first lines of code to understanding variables, lists, loops, and functions, you’ve got a solid foundation to build on.

The beauty of Python is that it grows with you. Whether you’re building simple scripts or diving into complex data science projects, Python’s intuitive design and versatility will keep you hooked. So keep practicing, keep experimenting, and soon enough, you’ll be writing programs that solve real-world problems.

Welcome to the world of Python — happy coding!

LibreOffice Calc for beginners: a complete guide to spreadsheets

2024-10-13T00:00:00+00:00

Are you new to LibreOffice Calc and eager to master spreadsheets? Don’t worry—you’re in the right place! Whether you’re a student, professional, or someone who just wants to organize their life better, LibreOffice Calc has got your back. It’s not just an alternative to Excel; it’s a powerful, open-source tool that can help you tackle everything from simple data tracking to complex calculations. Ready to dive in? Let’s break it down, step-by-step!

What Is LibreOffice Calc?

LibreOffice Calc is part of the LibreOffice suite, a free and open-source software that offers everything you’d expect from an office suite: word processing, presentations, and, of course, spreadsheets. Calc is comparable to Excel, but what sets it apart is that it’s completely free and backed by a strong community of developers from The Document Foundation.

But don’t be fooled by its price tag—or lack thereof. LibreOffice Calc is a heavyweight in the world of spreadsheets, and once you get a handle on its features, you’ll realize just how powerful it can be. Plus, it works on all platforms: Windows, macOS, Linux, and even Android!

Getting Started: Installing and Setting Up LibreOffice Calc

If you haven’t installed LibreOffice Calc yet, don’t sweat it. Head over to LibreOffice’s official site and download the latest version for your operating system. Once installed, simply open Calc from the LibreOffice suite, and you’re ready to roll.

Windows users: Works seamlessly on Windows 10, 11, and even older versions like Windows 7.
Mac users: LibreOffice Calc for Mac runs as smooth as butter, whether you’re on macOS Big Sur or Monterey.
Linux users: If you’re a Linux enthusiast, you’re in luck—LibreOffice Calc integrates effortlessly with most Linux distributions.

Now, with Calc installed and ready, let’s tackle some of the basics.

Navigating the Interface

LibreOffice Calc’s interface is clean and user-friendly. You’ll find all the essential tools at the top, just like you would in Excel.

Toolbars and Menus: These are packed with features, from basic formatting to complex data analysis tools. The most commonly used options are easily accessible.
Cells, Rows, and Columns: Just like Excel, Calc’s main workspace consists of cells arranged in rows and columns. The rows are numbered, and the columns are labeled with letters.
Formula Bar: This is where you enter your calculations and formulas. Click on a cell, type a formula, and hit Enter to calculate it.

Creating Your First Spreadsheet

Let’s start small. Imagine you’re creating a simple budget tracker for personal use. Open up a new sheet and follow these steps:

Column Titles: In Row 1, enter titles like Date, Description, Category, Amount, and Balance.
Data Entry: Start entering your expenses in the respective columns. For example, 2024-10-01 | Rent | Housing | 500.
Formulas: To keep a running total of your balance, click on the Balance cell in the first row and type =SUM(D2:D2). Drag the formula down the column to apply it to all rows.

Key Features Every Beginner Should Know

Here are some essential features to master as you get started with LibreOffice Calc:

1. Formulas and Functions

This is where spreadsheets shine! LibreOffice Calc supports a variety of functions like SUM(), AVERAGE(), IF(), and more. If you’ve used Excel, you’ll find Calc’s formula system is almost identical.

Example: If you want to calculate the total of a range of numbers, use the SUM() function. Just click on a cell and type =SUM(A1:A10) to add up everything from A1 to A10.
Pro Tip: You can use IF() functions to create conditional logic, like “If my balance drops below $100, highlight the cell in red.” That’s how you get smart with your data!

2. Formatting Cells

Making your data readable is as important as entering it. Select a cell or range of cells, right-click, and choose Format Cells. You can:

Adjust number formats (currency, date, percentages, etc.).
Add borders, background colors, and custom fonts.
Use conditional formatting to automatically change cell colors based on values.

3. Sorting and Filtering

Imagine you’re tracking a long list of transactions and want to see only those in a specific category, say “Housing.” Use AutoFilter:

Select your data range, go to Data > AutoFilter.
Click the small drop-down arrow that appears in your column titles and filter by category.

Need to sort your data? Select the range, and head to Data > Sort. You can sort by date, amount, or any other criteria.

4. LibreOffice Calc on the Cloud

Want to access Calc online? You can use LibreOffice Calc Online, a cloud-based version that allows you to edit your spreadsheets from anywhere. No more worrying about carrying files between devices. Just log in and get to work.

Excel to LibreOffice Calc: A Smooth Transition

If you’re transitioning from Excel, rest assured, most of your Excel files (XLS, XLSX) can be opened in LibreOffice Calc without any issues. However, for more complex spreadsheets with macros or advanced formatting, you may need to do some minor tweaking. Here’s how you can open an Excel file in Calc:

Open LibreOffice Calc.
Go to File > Open, select the Excel file, and open it.
If the formatting seems off, try saving it as an ODS file (the native LibreOffice format) to maintain better compatibility.

You can also convert LibreOffice Calc to Excel if you’re collaborating with others who prefer Microsoft’s software. Just save your file as XLS or XLSX (File > Save As).

Advanced Tips for When You’re Ready

Once you’re comfortable with the basics, it’s time to level up! Here are a few advanced tricks to explore:

1. Pivot Tables

Use pivot tables to summarize and analyze large datasets quickly. Go to Data > Pivot Table and explore this feature to create custom reports based on your data.

2. Charts and Graphs

Turn your data into visual insights by creating charts. Select your data and click on Insert > Chart. Calc supports bar charts, line graphs, pie charts, and more.

3. Macros

If you’re dealing with repetitive tasks, you can record macros to automate them. This is similar to Excel’s VBA scripting but uses LibreOffice’s built-in Basic language.

LibreOffice Calc on Mobile and Portable Versions

Need to access your spreadsheets on the go? LibreOffice Calc has Android support, allowing you to open and edit spreadsheets from your mobile device. Alternatively, you can use the portable version of LibreOffice Calc on a USB stick, so you can run it on any machine without installation.

Wrapping It Up

And there you have it—your beginner’s guide to LibreOffice Calc! From simple spreadsheets to advanced functions and data analysis, Calc offers a rich set of tools that rival paid alternatives like Excel. The best part? It’s free and open-source, so you can use it across platforms without worrying about licenses or subscriptions.

Whether you’re just tracking your personal budget or managing business data, LibreOffice Calc is your go-to spreadsheet solution. Now, go ahead and crunch those numbers like a pro!

Core PHP CRUD operations: a beginner's tutorial

2024-10-12T00:00:00+00:00

Creating dynamic web applications often involves interacting with a database. One of the foundational elements of web development is the ability to perform CRUD operations: Create, Read, Update, and Delete. This tutorial will guide you through core PHP CRUD operations, helping you understand how to manage data effectively in your applications. Whether you’re building a simple blog or a complex web app, mastering these operations will be essential for your development journey.

Prerequisites

Before diving into CRUD operations, ensure you have a basic understanding of the following:

HTML and CSS: To create user interfaces.
PHP: Knowledge of core PHP concepts.
MySQL: Basic understanding of relational databases and SQL queries.
XAMPP or MAMP: A local server setup to run your PHP scripts.

Setting Up Your Environment

Install XAMPP or MAMP: This will help you run PHP and MySQL on your local machine.
Create a Database: Open phpMyAdmin and create a database named crud_demo. Inside this database, create a table named users with the following structure:

CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    email VARCHAR(100) NOT NULL UNIQUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

This users table will hold the data for our CRUD operations.

Creating a PHP Script for CRUD Operations

1. Database Connection

Start by creating a config.php file to handle database connections.

<?php
$host = 'localhost';
$db_name = 'crud_demo';
$username = 'root'; // Change if your MySQL username is different
$password = ''; // Change if you have set a password

try {
    $conn = new PDO("mysql:host=$host;dbname=$db_name", $username, $password);
    $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
} catch (PDOException $e) {
    echo "Connection failed: " . $e->getMessage();
}
?>

2. Creating Data (Insert)

Next, create a create.php file to insert data into the users table.

<?php
include 'config.php';

if ($_SERVER['REQUEST_METHOD'] == 'POST') {
    $name = $_POST['name'];
    $email = $_POST['email'];

    $stmt = $conn->prepare("INSERT INTO users (name, email) VALUES (:name, :email)");
    $stmt->bindParam(':name', $name);
    $stmt->bindParam(':email', $email);
    
    if ($stmt->execute()) {
        echo "New user created successfully.";
    } else {
        echo "Error creating user.";
    }
}
?>

<form method="POST">
    Name: <input type="text" name="name" required>
    Email: <input type="email" name="email" required>
    <input type="submit" value="Create User">
</form>

3. Reading Data (Select)

Create a read.php file to display all users in the users table.

<?php
include 'config.php';

$stmt = $conn->prepare("SELECT * FROM users");
$stmt->execute();
$users = $stmt->fetchAll(PDO::FETCH_ASSOC);
?>

<table border="1">
    <tr>
        <th>ID</th>
        <th>Name</th>
        <th>Email</th>
        <th>Actions</th>
    </tr>
    <?php foreach ($users as $user): ?>
    <tr>
        <td><?php echo $user['id']; ?></td>
        <td><?php echo $user['name']; ?></td>
        <td><?php echo $user['email']; ?></td>
        <td>
            <a href="update.php?id=<?php echo $user['id']; ?>">Edit</a>
            <a href="delete.php?id=<?php echo $user['id']; ?>">Delete</a>
        </td>
    </tr>
    <?php endforeach; ?>
</table>

4. Updating Data

Now, create an update.php file to update user information.

<?php
include 'config.php';

$id = $_GET['id'];

if ($_SERVER['REQUEST_METHOD'] == 'POST') {
    $name = $_POST['name'];
    $email = $_POST['email'];

    $stmt = $conn->prepare("UPDATE users SET name = :name, email = :email WHERE id = :id");
    $stmt->bindParam(':name', $name);
    $stmt->bindParam(':email', $email);
    $stmt->bindParam(':id', $id);
    
    if ($stmt->execute()) {
        echo "User updated successfully.";
        header("Location: read.php");
        exit();
    } else {
        echo "Error updating user.";
    }
} else {
    $stmt = $conn->prepare("SELECT * FROM users WHERE id = :id");
    $stmt->bindParam(':id', $id);
    $stmt->execute();
    $user = $stmt->fetch(PDO::FETCH_ASSOC);
}
?>

<form method="POST">
    Name: <input type="text" name="name" value="<?php echo $user['name']; ?>" required>
    Email: <input type="email" name="email" value="<?php echo $user['email']; ?>" required>
    <input type="submit" value="Update User">
</form>

5. Deleting Data

Finally, create a delete.php file to handle user deletions.

<?php
include 'config.php';

$id = $_GET['id'];

$stmt = $conn->prepare("DELETE FROM users WHERE id = :id");
$stmt->bindParam(':id', $id);

if ($stmt->execute()) {
    echo "User deleted successfully.";
    header("Location: read.php");
    exit();
} else {
    echo "Error deleting user.";
}
?>

Understanding CRUD Operations

Create

The Create operation allows you to add new records to the database. In our example, we use an HTML form to gather user input and the PDO prepare and execute methods to insert data securely.

Read

The Read operation fetches and displays records from the database. We use a SQL SELECT query to retrieve user data, and we format it into an HTML table for easy viewing.

Update

The Update operation modifies existing records based on user input. We fetch the current data for a specific user, present it in a form, and update the record using a POST request.

Delete

The Delete operation removes records from the database. By passing the user ID via a URL parameter, we can execute a DELETE statement to remove the specified user.

Best Practices for PHP CRUD Operations

Use Prepared Statements: Always use prepared statements to protect against SQL injection.
Validate User Input: Implement server-side validation for user input to ensure data integrity.
Implement Error Handling: Use try-catch blocks to gracefully handle potential errors.
Organize Your Code: Keep your code organized by separating concerns (e.g., database connections, business logic, and presentation).
Follow Security Best Practices: Sanitize inputs, hash passwords, and secure your application against common vulnerabilities.

Conclusion

In this tutorial, we’ve covered the essential core PHP CRUD operations, demonstrating how to create, read, update, and delete records in a MySQL database. By understanding these foundational operations, you’ll be well-equipped to build dynamic web applications.

Remember that mastering CRUD operations is just the beginning. As you advance in your PHP journey, explore additional concepts such as AJAX integration, data validation, and security measures to enhance your applications further.

With practice and patience, you’ll be able to implement complex functionalities and create robust web applications that meet various user needs. Happy coding!

Common PHP errors: solutions to frequently encountered issues

2024-10-09T00:00:00+00:00

PHP is a powerful scripting language widely used for web development, but like any language, it’s easy to run into errors that can be frustrating to debug. While some errors are simple and easy to fix, others may be a little more complex. This article covers some of the most common PHP errors and offers solutions to help you resolve them quickly.

1. Syntax Errors

Problem:

A syntax error occurs when the PHP interpreter encounters code that doesn’t conform to the expected structure. These are the most basic types of errors and often result in the dreaded Parse error: syntax error, unexpected token message.

Common Causes:

Missing semicolons (;)
Unmatched parentheses, curly braces, or brackets
Incorrect use of quotation marks
Misspelling keywords

Example:

echo "Hello World" // Missing semicolon

Solution:

Double-check your code for missing or extra punctuation. Make sure all your opening and closing parentheses, brackets, and quotes match.

echo "Hello World"; // Fixed

2. Undefined Variable Error

Problem:

An “undefined variable” error occurs when you try to use a variable that has not been initialized. PHP will throw a Notice: Undefined variable error in this case.

Example:

echo $username; // Undefined variable

Solution:

Ensure that the variable is initialized before using it in your code. You can also suppress this notice by checking if the variable is set using isset().

if (isset($username)) {
    echo $username;
} else {
    echo "No username provided";
}

3. Fatal Error: Call to Undefined Function

Problem:

This error occurs when you attempt to call a function that hasn’t been defined. It could happen because you misspelled the function name or forgot to include the necessary file containing the function.

Example:

myFunction(); // Undefined function

Solution:

Ensure that the function is properly defined or included in your script. Also, check for typos in the function name.

function myFunction() {
    echo "Hello World!";
}

myFunction(); // Fixed

4. Headers Already Sent

Problem:

This error occurs when PHP tries to modify headers (e.g., with header() or setcookie()) after output has already been sent to the browser. The error message typically looks like this: Warning: Cannot modify header information - headers already sent by...

Example:

echo "Some output";
header("Location: /newpage.php"); // Causes error because output was already sent

Solution:

Ensure that no output (including whitespace or BOM) is sent before the header() function is called. If you need to redirect the user, make sure the header() is called before any output is generated.

header("Location: /newpage.php"); // This must appear before any echo or print statements
exit();

5. Incorrect Permissions

Problem:

Permission errors occur when your PHP script does not have the proper read or write permissions to access files or directories. You might see errors like Warning: fopen(/path/to/file): failed to open stream: Permission denied.

Solution:

Check the file and directory permissions. Typically, web server users should have read permissions for files and write permissions for directories where uploads or file manipulations occur. Use the following command to adjust permissions:

chmod 755 /path/to/directory
chmod 644 /path/to/file

Note: Be cautious when setting permissions, as overly permissive settings can pose security risks.

6. Memory Limit Exhausted

Problem:

When PHP runs out of allocated memory, you’ll see a Fatal error: Allowed memory size of X bytes exhausted error. This happens when a script uses more memory than the limit set in php.ini.

Solution:

You can increase the memory limit temporarily by adding the following line to your PHP script:

ini_set('memory_limit', '256M'); // Adjust as needed

Alternatively, you can permanently increase the memory limit in the php.ini file:

memory_limit = 256M

Make sure to optimize your code to reduce memory usage where possible.

7. MySQL Connection Error

Problem:

Connecting to a MySQL database can sometimes fail, resulting in an error like Fatal error: Uncaught mysqli_sql_exception: Access denied for user 'username'@'localhost'.

Common Causes:

Incorrect database credentials (hostname, username, password, database name)
The MySQL server is not running
Incorrect PHP MySQL extension (e.g., using mysql_connect() instead of mysqli_connect())

Solution:

Ensure that your credentials are correct and that the MySQL server is running. Also, make sure to use the appropriate connection function. Here’s a correct example using mysqli_connect():

$mysqli = new mysqli('localhost', 'username', 'password', 'database');

if ($mysqli->connect_error) {
    die("Connection failed: " . $mysqli->connect_error);
}

8. File Upload Errors

Problem:

File uploads often fail due to improper settings or file size limitations. You may encounter errors like UPLOAD_ERR_INI_SIZE or UPLOAD_ERR_FORM_SIZE.

Solution:

Check and adjust the following php.ini settings as needed:

file_uploads = On
upload_max_filesize = 10M
post_max_size = 12M

Also, make sure your form tag has the correct enctype attribute:

<form action="upload.php" method="post" enctype="multipart/form-data">
    <input type="file" name="file">
    <input type="submit" value="Upload">
</form>

9. Undefined Index/Offset

Problem:

This notice occurs when you try to access an array element that doesn’t exist, causing a Notice: Undefined index or Notice: Undefined offset error.

Example:

echo $_POST['username']; // Undefined index if 'username' is not in the form data

Solution:

Always check if the array key exists before trying to access it. Use isset() or array_key_exists() to prevent this error.

if (isset($_POST['username'])) {
    echo $_POST['username'];
} else {
    echo "Username not provided.";
}

10. Class Not Found

Problem:

PHP throws a Fatal error: Class 'ClassName' not found error when you try to instantiate a class that hasn’t been defined or included properly.

Solution:

Ensure that the file containing the class is included using require() or include(). Alternatively, use PHP’s spl_autoload_register() function to automatically load class files.

spl_autoload_register(function ($class_name) {
    include $class_name . '.php';
});

$object = new ClassName();

11. Maximum Execution Time Exceeded

Problem:

If your PHP script takes too long to execute, you may encounter the Fatal error: Maximum execution time of X seconds exceeded error. This usually happens when working with large datasets or external API calls.

Solution:

You can increase the maximum execution time temporarily with:

set_time_limit(300); // Extends to 300 seconds (5 minutes)

To set it globally, adjust the max_execution_time directive in the php.ini file:

max_execution_time = 300

PHP errors are inevitable, but knowing how to tackle the most common ones can save you a lot of debugging time. Whether it’s a syntax issue, database connection problem, or file permission error, understanding the root cause and solution is key to becoming a proficient PHP developer.

By following the guidelines in this article, you should be able to identify and resolve these issues effectively. Keep your error reporting enabled during development to catch these errors early and ensure smoother coding!

Unlocking speed: the power of indexing in database performance

2024-10-08T00:00:00+00:00

In today’s data-driven world, the performance of databases is crucial for applications, businesses, and users alike. As databases grow in size and complexity, the need for efficient data retrieval becomes paramount. One of the most effective strategies for enhancing database performance is indexing. This article explores the role of indexing in database performance optimization and how it significantly improves query performance.

Understanding Indexing

What is an Index?

An index in a database is a data structure that improves the speed of data retrieval operations on a database table. It functions similarly to an index in a book, allowing the database management system (DBMS) to quickly locate and access the desired records without having to scan every row in a table.

How Indexes Work

Indexes work by creating a separate structure that holds the key values and pointers to the actual data rows. When a query is executed, the DBMS can use the index to quickly locate the rows that match the search criteria, minimizing the amount of data that needs to be scanned.

Types of Indexes

There are several types of indexes, each with its unique characteristics and use cases. The most common types include:

1. B-Tree Indexes

B-Tree (Balanced Tree) indexes are the most widely used index type in relational databases. They store data in a hierarchical structure, enabling efficient retrieval of sorted data. B-Tree indexes are particularly effective for range queries and equality searches.

2. Hash Indexes

Hash indexes use a hash table to store data, mapping keys to their corresponding values. They provide extremely fast lookups for equality comparisons but are not suitable for range queries.

3. Bitmap Indexes

Bitmap indexes use bit arrays to represent the existence of values in a column. They are particularly useful for low-cardinality data (columns with a limited number of distinct values) and can significantly reduce storage requirements and improve performance for complex queries.

4. Full-Text Indexes

Full-text indexes are designed for searching text within large volumes of unstructured data. They enable efficient searching and retrieval of documents based on keywords, phrases, and other criteria.

The Benefits of Indexing

1. Faster Query Performance

The primary benefit of indexing is improved query performance. By using indexes, the DBMS can reduce the number of data pages it must read, leading to faster response times for queries. For example, a search that could take seconds without an index may take milliseconds with one.

2. Reduced I/O Operations

Indexes minimize the number of disk I/O operations required to retrieve data. By allowing the DBMS to quickly locate data pages, indexes significantly reduce the workload on the disk subsystem, leading to improved overall performance.

3. Enhanced Sorting and Filtering

Indexes enable efficient sorting and filtering of data. When executing queries that involve sorting or filtering, the DBMS can leverage indexes to access the necessary data more quickly than performing a full table scan.

4. Improved Join Performance

Indexes can enhance the performance of join operations by allowing the DBMS to quickly locate matching rows in different tables. This is especially beneficial for large datasets where join operations can otherwise become a significant performance bottleneck.

5. Support for Unique Constraints

Indexes can enforce uniqueness constraints on a column or a combination of columns. This ensures that duplicate values are not allowed, helping maintain data integrity within the database.

Considerations When Implementing Indexes

While indexing offers numerous benefits, it’s essential to approach it with care. Over-indexing or improperly indexed tables can lead to several issues:

1. Increased Storage Requirements

Indexes consume additional disk space. Each index created increases the amount of storage required for the database. It’s essential to balance the benefits of improved performance against the additional storage costs.

2. Slower Write Operations

Indexes can slow down insert, update, and delete operations because the index needs to be maintained with every modification to the data. This is particularly important for applications with high write workloads, where the overhead of maintaining indexes can become significant.

3. Choosing the Right Columns

Not all columns benefit equally from indexing. Careful consideration must be given to which columns to index based on the types of queries being executed. Columns frequently used in WHERE clauses, JOIN conditions, and ORDER BY statements are good candidates for indexing.

4. Monitoring and Maintenance

Indexes require regular monitoring and maintenance. As data changes, indexes can become fragmented, leading to reduced performance. Regularly rebuilding or reorganizing indexes can help maintain optimal performance.

Best Practices for Indexing

To maximize the benefits of indexing while minimizing potential drawbacks, consider the following best practices:

1. Analyze Query Patterns

Before creating indexes, analyze the query patterns in your application. Focus on optimizing queries that are executed frequently and have high execution times.

2. Use Composite Indexes

When multiple columns are often used together in queries, consider creating composite indexes. These indexes can improve performance for queries that filter based on multiple columns.

3. Regularly Review and Optimize Indexes

Periodically review your indexes to ensure they are still providing value. Remove unused or redundant indexes that may be consuming resources without delivering performance improvements.

4. Limit the Number of Indexes

Avoid over-indexing. Focus on creating indexes that will have a significant impact on query performance while keeping the total number of indexes manageable.

5. Monitor Performance Impact

Continuously monitor the impact of indexes on database performance. Use database profiling tools to evaluate how indexes affect query performance and make adjustments as needed.

Indexing is a powerful tool for optimizing database performance, significantly enhancing query speed and efficiency. By understanding the various types of indexes and their benefits, database administrators and developers can make informed decisions about when and how to implement indexing strategies. With careful planning and regular maintenance, indexing can lead to a more responsive and efficient database, ultimately improving the overall user experience.

PHP libraries for PDF handling: evaluation and use case guide

2024-10-06T00:00:00+00:00

PDF generation and processing is a common requirement in many web applications, especially for generating invoices, reports, or documents dynamically. PHP provides various libraries to handle PDF creation, manipulation, and extraction. In this article, we will evaluate the most popular PHP libraries for PDF handling, breaking down their pros and cons, and advising when each library is best suited for specific use cases.

1. FPDF

FPDF is one of the most basic and lightweight libraries for creating PDF files with PHP. It requires no external dependencies and allows developers to generate PDFs from scratch using a simple API.

Pros:

Lightweight: FPDF is minimalistic, making it easy to set up and use for basic PDF generation.
Free: It’s open-source and free to use, suitable for small projects.
Simple API: The syntax is intuitive and easy for developers to understand, making it ideal for beginners.
Customizability: Gives control over text placement, fonts, images, and shapes.

Cons:

No HTML/CSS support: FPDF does not support converting HTML or CSS into PDF, making it harder to design complex layouts.
Limited Features: While it’s good for basic tasks, it lacks advanced features like embedding fonts, handling complex table layouts, and supporting Unicode.

Best Use Cases:

Generating simple invoices, tickets, or certificates.
Projects that don’t require complex layouts or HTML-to-PDF conversion.

2. TCPDF

TCPDF is a robust, feature-rich library that builds on the simplicity of FPDF but provides more advanced capabilities like support for HTML, Unicode, and complex layouts.

Pros:

Feature-Rich: TCPDF supports HTML and CSS, including tables, images, and complex document structures.
Unicode Support: It has native support for UTF-8, making it ideal for multilingual projects.
No Dependencies: Unlike other libraries that rely on external tools, TCPDF is standalone.
Barcodes and QR codes: It has built-in support for generating barcodes and QR codes, a useful feature for product labels or tickets.

Cons:

Performance: TCPDF is slower and more memory-intensive than FPDF due to its extensive features.
Steep Learning Curve: Its vast array of features can be overwhelming for new users.
Large Files: PDF files generated by TCPDF tend to be larger compared to those generated by FPDF or other libraries.

Best Use Cases:

Applications that require multilingual or RTL text support.
Generating complex documents with images, tables, and different fonts.
Projects that need barcodes or QR codes within PDFs.

3. mPDF

mPDF is based on FPDF and HTML2PDF and is designed for converting HTML content into PDFs. It provides a good balance between simplicity and features, offering an easy way to convert web pages into PDFs.

Pros:

HTML/CSS to PDF Conversion: mPDF excels at converting complex HTML structures (with CSS) into PDF, making it ideal for web-based PDFs.
Unicode and RTL: It supports multilingual content, including right-to-left (RTL) scripts like Arabic.
Embedded Fonts: Supports embedding custom fonts within the PDF, adding to design flexibility.
Ease of Use: It’s relatively easy to convert a web page into a PDF with minimal configuration.

Cons:

Performance: While fast for small documents, mPDF can become slow with large or complex HTML layouts.
Memory Usage: mPDF consumes more memory than some other libraries when handling large documents.
Limited Features: While it works well with HTML/CSS, it lacks advanced PDF manipulation features (e.g., extracting text from PDFs).

Best Use Cases:

Converting web pages (HTML + CSS) directly into PDF.
Projects that require custom font embedding and support for multiple languages.
Ideal for invoices, reports, or forms that are originally designed as web pages.

4. Snappy (wkhtmltopdf)

Snappy is a wrapper for the wkhtmltopdf command-line tool that uses WebKit to render HTML as PDF. It’s known for producing high-quality, accurate PDFs from complex HTML/CSS structures.

Pros:

High-Quality Output: Snappy (wkhtmltopdf) delivers precise and accurate rendering of complex layouts, including advanced CSS and JavaScript.
JavaScript Support: It can execute JavaScript, making it possible to include dynamic content (e.g., charts) in your PDF.
Speed: wkhtmltopdf is relatively fast compared to libraries like TCPDF and mPDF for large documents.
Command-line Utility: Allows for batch processing via the command line for server-side applications.

Cons:

Requires External Tool: It depends on the wkhtmltopdf tool, which must be installed on the server.
Limited PDF Manipulation: Snappy is great for converting HTML to PDF, but lacks advanced features like merging or splitting PDFs.
Resource-Heavy: It can be resource-intensive, especially when generating PDFs from very complex HTML layouts.

Best Use Cases:

Projects that need pixel-perfect HTML-to-PDF conversion.
Applications where you need to convert large reports or invoices from HTML to PDF quickly.
Dynamic web content (with JavaScript) that needs to be rendered as a PDF.

5. PDFParser

PDFParser is a powerful library for parsing and extracting content from existing PDF files. It focuses on analyzing and extracting text, images, and metadata rather than creating new PDFs.

Pros:

Text Extraction: It efficiently extracts text, images, and metadata from existing PDFs.
Search and Indexing: Useful for indexing and searching large collections of PDFs.
Built on TCPDF: Leverages TCPDF’s capabilities for more complex parsing tasks.

Cons:

No PDF Creation: PDFParser is specifically for extracting content, not creating or manipulating PDFs.
Complex Documents: While it works well with simple documents, it can struggle with highly complex or encrypted PDFs.

Best Use Cases:

Projects that involve PDF document indexing, searching, or text extraction.
PDF content analysis or metadata extraction tasks.

6. SetaPDF

SetaPDF is a commercial-grade PHP library for professional-level PDF processing. It offers features like form filling, signing, and advanced manipulation of PDF files.

Pros:

Enterprise Features: Includes professional-level features like digital signatures, form filling, and annotations.
High Performance: Optimized for enterprise environments with high-volume PDF processing.
Security: Offers advanced encryption and security features for PDF files.
Rich API: Comprehensive API for advanced PDF manipulation.

Cons:

Commercial License: SetaPDF is a paid library, which might be a barrier for smaller projects or individual developers.
Complexity: The advanced feature set requires a steep learning curve and in-depth understanding of PDFs.

Best Use Cases:

Enterprise applications that need secure, high-performance PDF generation, filling, or signing.
Projects involving sensitive documents like contracts, legal papers, or certifications.

7. FPDI

FPDI is an extension to FPDF and TCPDF that allows you to import pages from existing PDFs and use them in newly generated PDFs. It’s perfect for modifying and merging PDFs.

Pros:

PDF Importing: Allows you to import, reuse, and manipulate existing PDF pages.
Works with FPDF and TCPDF: Combines well with either library to provide enhanced functionality.
Simple to Use: If you’re already using FPDF or TCPDF, FPDI is easy to integrate.

Cons:

No PDF Creation: FPDI is designed for manipulating and merging PDFs, not for creating new ones from scratch.
Limited Editing: It’s great for importing and merging but limited for complex editing of PDF content.

Best Use Cases:

Applications where merging multiple PDFs is required (e.g., combining invoices or reports).
Projects that need to modify existing PDFs, like adding a cover page or watermark.

8. PDFMerger

PDFMerger is a small, easy-to-use library for merging PDF files. It simplifies the process of combining multiple PDF files into one document.

Pros:

Simplicity: Very easy to use for merging PDFs.
Quick Integration: You can quickly add this library to an existing project.
Lightweight: Minimal dependencies and configuration needed.

Cons:

Limited Functionality: Focuses purely on merging; does not support any other PDF manipulation.
No Editing: You can merge PDFs but cannot modify their contents.

Best Use Cases:

Projects that require merging multiple PDFs into a single file.
Use cases like bundling invoices, contracts, or reports.

9. Dompdf

Dompdf is one of the most popular PHP libraries for converting HTML and CSS to PDF. It’s a powerful library that uses a layout engine based on HTML5 and CSS3, allowing developers to generate PDFs from existing web pages or dynamically built HTML documents.

Pros:

HTML/CSS Support: Dompdf supports most HTML5 tags and CSS3 styles, making it suitable for converting complex web pages into PDFs.
Ease of Use: Very straightforward for developers familiar with HTML and CSS, as it allows you to render PDFs directly from existing templates.
Open Source: Dompdf is open-source and free, with a large community for support.
Font Embedding: Allows you to embed custom fonts within the PDF for more design flexibility.
Basic JavaScript Support: Supports basic JavaScript but is somewhat limited compared to other libraries like Snappy (wkhtmltopdf).

Cons:

Performance: Dompdf can be slow when dealing with large or complex HTML documents, especially those with many images or heavy CSS.
Limited Advanced Features: Compared to libraries like TCPDF or mPDF, Dompdf lacks advanced features like encryption, digital signatures, or form fields.
Inconsistent CSS Rendering: While it supports CSS3, it can struggle with some advanced CSS properties, especially for responsive design elements.
Memory-Intensive: Processing large PDFs can lead to high memory usage and longer generation times.

Best Use Cases:

Web-to-PDF Conversion: Perfect for converting web pages into PDF, such as online invoices, reports, or newsletters that are built in HTML.
Small to Medium-Sized Projects: Works well for web applications that generate medium-sized PDFs (e.g., e-commerce invoices, user manuals).
PDFs with Custom Designs: Ideal for generating PDFs with custom fonts, layouts, and styling, particularly if the source content is already styled with HTML/CSS.

When to Use Dompdf:

HTML-to-PDF Conversion: Like mPDF, Dompdf is designed specifically for converting HTML and CSS to PDF, making it the go-to choice for web applications that need to render dynamic HTML pages into PDF format.
Simple Projects with Moderate HTML: It’s ideal for medium-complexity documents that require basic to moderate HTML/CSS rendering. However, if your project involves large documents, high resolution images, or complex layouts, you may run into performance bottlenecks.
Projects with Embedded Fonts: Dompdf’s ability to embed custom fonts makes it a good choice for documents where typography is essential, such as branded reports or certificates.

In summary, Dompdf is a great option for converting existing HTML content into PDF format when you have a simple or moderately complex design and don’t need advanced PDF features like encryption or form processing. However, for larger documents or projects that require more sophisticated PDF capabilities, you may want to consider alternatives like TCPDF, mPDF, or Snappy.

When to Use Each Library

FPDF: Use when you need a lightweight solution for creating basic PDF documents from scratch.
TCPDF: Ideal for generating complex, feature-rich PDFs with support for HTML, Unicode, and custom fonts.
mPDF: Best for converting web pages (HTML + CSS) into PDFs. Great for forms, invoices, and reports designed for the web.
Snappy (wkhtmltopdf): Go-to for pixel-perfect conversion of complex HTML/CSS to PDFs, especially for reports or invoices.
PDFParser: Suitable for extracting text and metadata from existing PDF files.
SetaPDF: The choice for enterprise-level projects requiring secure PDF handling, digital signatures, and form processing.
FPDI: Use when you need to import or manipulate existing PDFs within new PDF files.
PDFMerger: Perfect for merging multiple PDFs into one document, ideal for report bundling.
Dompdf: For converting HTML/CSS to PDF in small to medium-sized projects with simple to moderate layouts. It’s ideal for web applications needing dynamic PDFs with custom fonts and designs.

Each library has its strengths and weaknesses, and the choice depends heavily on the specific requirements of your project. For HTML-to-PDF conversion, mPDF or Snappy are excellent choices. If you’re dealing with secure enterprise documents, SetaPDF is your best bet. For basic tasks, FPDF offers a straightforward and lightweight solution.

Who exactly I am as a soul? - A crash course in Advaita Vedanta

2024-09-30T00:00:00+00:00

“Who am I?” is perhaps the most sought after question by humankind since time immemorial.

Do you even think that you have a soul or consciousness as separate entity than the physical body and biological brain? If not, you may be an absolute materialist. On the other hand, do you think you live in a transient world of Matrix or a long-term dream and the reality will be faced only when you go out of it and meet the creator? In this case, you’re an absolute spiritualist. In Indian philosophy or Darshan Shastras, these are the two polar opposites (material and spiritual), the former represented by Charvaka and latter by Advaita Vedanta schools of thoughts. Even Buddhism and Jainism generally lean towards spiritual than material.

The ideal approach to life is to balance out these philosophies and not put all the money in any one stock! However, the spiritual and material tendencies of humans also depend a lot on circumstances. An ambitious and materialistically wealthy person in the youth phase of their life will put more stock in materialism as that is paying off right now. On the other hand, someone who has aged beyond 40-50s or someone who is in a declining phase and facing difficulties might incline more towards spirituality.

Even if you can’t answer that question or ever thought about it, everyone does have a “working theory” of sorts about who they are in their minds. And that working theory determines literally everything in their life - their choice of food, clothes, lifestyle, attitude towards wealth, expected salary and even political inclination!

As per Advaita theory, deep within the subconscious mind, there is an inner urge in every soul to know itself, it’s literally crying out “Who am I?”. The first cry of an infant when it lands on the Earth is reflective of that. Having failed to get an answer, the soul then attaches to whichever identity it finds around, the identities of relationships, career, caste, creed, religion, ideology, etc. Latching to these identities then becomes a substitute for seeking that timeless answer, in fact it becomes an objective in itself, driven by desires and emotions through life.

The moment you answer that question, and actually realize what that answer is, realization or nirvana is achieved and you become free from the Matrix. But the answer itself isn’t the important part of the quest, the real deal is to be aware of these sheaths and layers of identities and destroy them. As long as the soul is trapped in the sheaths, it will be driven by its own emotions and face its karma or consequences thrown by the material world, as if on an autopilot. Even after death, the same story repeats in another part of the Matrix (maybe another planet on another galaxy, or even centuries apart on this same planet).

The Gita describes three kinds of tapas or depressions - Adi bhautik (material/wealth related depressions), Adi daivik (depressions arising out of fate or acts of God like pandemic, disease, accidents, etc.) and finally Adhyatmic (spiritual depression). If a soul has achieved everything they could in terms of material wealth and they’re still unhappy or depressed, it only means they’re now ready to deal with that final or ultimate question in spirituality.

Sri Adi Shankaracharya, the author of Advaita Vedanta, has composed a beautiful Sanskrit poem called Nirvana Shatakam which beautifully reflects on each one of these material sheaths or layers one by one. The soul declares and distances itself from each one of these sheaths like mind, intellect, ego, relationships, etc.

In an attempt to answer the ultimate question, Adi Shankaracharya declares every now and then “Shivoham Shivoham” (I am Shiva or God) in that poem. This has been one of the most controversial aspects of advaita vedanta, along with his other famous phrase “Aham brahmasmi” which also means “I am God”, and gets criticized a lot. Being born in a Dvaita tradition family, I’ve often heard this criticism by many - “How can one declare themselves as God?”. The Dvaita school asserts on the separation of Jiva (soul), Brahma (God) and Jada (material), and advocates bhakti instead as the path towards liberation.

Indeed, declaring oneself as God seems foolhardy or even lunatic in a fully materialistic world, where “you” and “me” are considered two different souls. In fact, this is the world which we daily observe. But what if this was essentially a “Matrix” or transient world or one of a long-term dream as shown in the movie “Inception”? In that case, every single soul is just a part of the “greater dream” along with all the matter and energy that is. And thus, every soul can be referenced as a God or creator since they’re inside the creator’s dream? This dream or matrix is called Maya. This idea isn’t new, that our world could be a Matrix like thing is also a basic tenet of Gnosticism which was a collection of religious and philosophical ideas among early Christian and Jewish sects.

Even Buddhism is pretty much based on the idea that the material world is transient and not substantial, the Mahayana branch goes even further and talks of a Pure Land, the better version of this Matrix where folks wanting to achieve liberation can go after this life ends.

Other factors that support a “fluxed” Matrix like reality:

The Economic theory of diminishing marginal utility. Nothing in this world brings yatharth or lasting happiness. Even the best of SUVs, the best brand of alcohol, the best gated community will give you dopamine only till a certain point. Post that, the dopamine resistance sets in and so does the Adhyatmic tapa or Spiritual depression.
The Double Slit Experiment. This experiment in Quantum Physics shows that particles suddenly become waves depending on whether they’re being observed or not. This isn’t a sign of a yatharth or stable world, it’s a sign of a transient or matrix world.
Human life itself is unpredictable. The built-in emotions like hope and greed keep driving the samsara or life but if you think about it, data shows that all of that zeal may not stand the test of time. We barely managed to escape a pandemic few years ago, India is just one political turmoil away or one pandemic away from massive destruction of civilization. This doesn’t mean we should stop doing our work, but the over zealousness of negative emotions like greed, jealousy, crony capitalism, etc. can be certainly contained.
Experiential or subjective truth counts. Each time I listen to the Nirvana Shatakam poem and the responses being given to the soul’s yearning, I feel the veil of Matrix being lifted just a little bit. But this is just my own subjective experience. In either case, it’s a beautiful poem and I recommend listening to it at least once.

Seven most aesthetically pleasing fonts for your web apps

2024-06-15T00:00:00+00:00

Without Google Fonts, our typography would be in limbo! Now, I’m neither a web designer nor have an eye for the pixel perfect and aesthetic design which a pro designer usually does. Instead, my usual workflow is to cobble together web components like bootstrap, jquery, codeigniter, etc. and try to build something useful out of it.

As for fonts, I happen to use the open source Google Fonts of which there are a zillion varieties. But what I’ve found over the years is that a few of them outshine the rest when it comes to actual web and mobile applications (as opposed to other kind of websites such as static sites and blogs).

Rubik is one of my favorite fonts when it comes to app development. The combination of Rubik font and a comprehensive CSS framework like Bootstrap is quite powerful and formidable. You'll see many professional open source apps using this font in the wild. The popular software ERPNext used it extensively until very recently, only now they've switched to Inter.

Inter is yet another aesthetically pleasing font, especially for web applications. Its fonts page says, "Inter is a variable font family carefully crafted & designed for computer screens." I've largely found this font quite readable on app screens.

Mulish is the Google's open source equivalent of Adobe's original Muli font. It looks quite elegant and aesthetic on web apps. I've used the Mulish font in my open source side project called Comment Monk, a self-hosted comments CMS for static blogs.

Roboto is the most downloaded font on the internet. According to its fonts page, "Roboto has a dual nature. It has a mechanical skeleton and the forms are largely geometric. At the same time, the font features friendly and open curves.".

Open Sans is often touted as the open source humanist sans-serif typeface. Designed by Steve Matteson under commission from Google and published in 2011, Open Sans is a successor to the other popular font called Droid Sans which was specifically crafted for android mobile devices.

Lato is yet another humanist sans-serif typeface designed by Łukasz Dziedzic. Published in 2010, the name Lato is Polish translation for "summer" and indeed, one could almost feel that warmth in this font.

Libre Baskerville is yet another elegant web font. Based on the American Type Founder's Baskerville from 1941, Libre Baskerville is optimized for body text (approximately 16px) and looks beautiful on typical web apps as well.

To include a font from Google Fonts in your app, just link to their stylesheet with the “family” querystring parameter set to whatever you want as I’ve done on this very page. If there are multiple, you can separate them using the filter character (|).

<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Rubik|Inter|Mulish|Roboto|Libre+Baskerville|Open+Sans|Lato">

@todo: Create a placeholder app on github.io using bootstrap components to test these fonts.

Introducing Comment Monk: simple comment hosting system for static blogs and websites

2024-06-07T00:00:00+00:00

I wanted to implement a comment hosting system for my static blog—something as simple as the basic WordPress.org commenting feature, with fields for the user’s name, website, etc. No complicated logins, sign-ups, or third-party platforms. The user reads your blog, posts a comment, and you approve it from the backend (or, alternatively, it gets auto-approved, and you receive an email notification). As simple as that!

Since GitHub Pages doesn’t provide backend PHP scripting, I had to develop an entire backend app along with a frontend ECMAScript (JavaScript) script that could be plugged into a div block at the end of a blog post—typically the space reserved for comments. Comment-Monk is the result of that effort. I’ve made this app open-source and hosted it on GitHub so that it can be used by as many folks as possible.

To use this app on your own static blog, simply download the repo and deploy it to a PHP web hosting service. It’s a lightweight script with an SQLite backend, intentionally kept small enough to be deployed on one of those affordable (even free) PHP hosting services.

Once you start the app, you’ll be taken to the Install page, where you can register with your details and credentials, which you’ll use to log in as a superuser. It also asks for the website or domain of your blog where you’ll host the commenting system.

Once logged in and on the homepage, you can see this screen where you can view and manage comments. Right now, you can only view and delete comments, but more features are on the way in upcoming versions. You can also set your user preferences from the “Actions” menu in the top right.

Most importantly, you can click on the “Client Snippet” button, which will guide you through implementing the frontend HTML code to embed the comments.

Once you’ve done that, your static blog should look something like this:

At the place where you added the script tag, you should be able to see a comment block displaying all existing comments along with a submission form for posting new ones.

Your audience can read your content and post comments. The backend validates the URI (Uniform Resource Identifier), and if a registered domain is found, it creates an entry for that comment, which will be reflected on the administrator’s (YOUR) dashboard. I think this is about as simple as it can get!

The comment block has a very basic and minimal design by default, but you can fully customize it by editing the /static/cm-client.css on the backend, which is used for styling.

I hope you find this system useful for your static blog. If you encounter any issues, don’t hesitate to raise them on the GitHub tracker. Happy coding!

Edit: 2024-11-07

While Comment-Monk was a great academic exercise, I realized through this experiment that hosting and maintaining your own custom solution isn’t as trivial for an indie tech blogger as it may seem. To be honest, it’s not even necessary. Between the complexity of maintaining a custom comment system and the ad-ridden, privacy-unfriendly platforms like Disqus, I found Giscus to be a good middle ground. That’s why I made the switch.

How to fix "silent error handling" of PDO errors after upgrading to PHP 8

2024-06-07T00:00:00+00:00

PHP 8.x is filled with hidden land mines which can suddenly trip the unaware PHP coder, especially when porting from legacy frameworks like CodeIgniter or CakePHP to PHP 8.x systems. One such issue is PDO’s default exception handling method.

Prior to PHP 8, PDO’s default exception handling mode was PDO::ERRMODE_SILENT which means our apps deployed on production never expected any actual PDO Exceptions, they instead handled database errors by looking up the error code after the fact with something like CodeIgniter’s $CI->db->error() function, for instance.

But starting from PHP 8.0, the default exception handling mode for PDO is changed to PDO::ERRMODE_EXCEPTION. This means most PHP apps developed with the prior mindset can expect sudden or subtle bugs depending on how they were coded!

If you usually keep PHP’s display_errors INI setting to 0 to avoid unnecessary depreciation notices on your production, then you are in for a great nasty surprise on PHP 8.x! Because if any database error occurs, you will never come to know of it. Your usual error code based handling won’t work here as PDO uses the exception handling mode by default. Which means PDO still raises this exception, only it won’t be displayed anywhere due to your display_errors INI setting. It will silently crash your script execution with no output and a 500 HTTP status code.

In one of the companies where I contract, this is exactly what happened recently. The following CI3 code is highly dependent on error code based checking - which is done inside the db_error() helper function. But the sad thing is that the interpreter will never reach that point as $this->db->query will throw an exception and halt the process immediately.

$this->db->query("insert into schedule(emp_id, place_id, sch_date, notes) values (?,?,?,?)", $data);
if ($err = db_error()) {
	$r['status']  = 'failure';
	$r['message'] = $err;
} else {
	$r['status']='success';
	$r['message'] = 'Successfully inserted record.';
}
echo json_encode($r);

I had to debug by placing multiple error_log() statements to find out what’s going on here! Needless to say, even though Exception is the recommended way of handling PDO hiccups, we can’t change the whole app and its workflow for that alone, the cost would be too prohibitive for larger PHP apps like these. What we did instead was updated CodeIgniter’s database configuration and changed the PDO’s error handling option back to PDO::ERRMODE_SILENT instead:

# /application/config/database.php
#..
#..
$db['default']['options'] = array(PDO::ATTR_ERRMODE => PDO::ERRMODE_SILENT);

If you don’t want to do this but take the righteous path of using PDO Exceptions instead, then make sure you report and display ALL errors (or at least all barring depreciation notices):

//error_reporting(E_ALL);
error_reporting(E_ALL & ~E_NOTICE & ~E_DEPRECATED);
ini_set('display_errors', 1);

After that, you can handle PDO Exceptions in your code normally. But the important thing is that you’ll now be able to see the errors and know what’s going on. They won’t be treated silently like before.

Happy Coding!

How to install composer packages locally or offline

2024-06-01T00:00:00+00:00

The usual way of installing composer packages with composer require <foo> is the standard. But occasionally, we might want to install the packages from our local computer to save Internet bandwidth or some other reason.

Unlike Python’s pip, PHP’s composer doesn’t offer us a straightforward path to install from an archived package file here by running something like pip install -i <package-file-name>. However, there are two paths or workarounds to achieve this as follows:

[1] The artifact way.

This is the more preferable way, especially if your goal is to store the composer package files (*.zip) on your storage drive for long term use. But do note that if you go this route, you will be responsible for installing any dependency packages your app may have.

To do this, just edit your app’s composer.json file and add your storage path in the repositories section:

"repositories": [
  {
    "type": "artifact",
    "url": "./packages"
  }
]

The ./packages could be any arbitrary local path on your computer. When you specify this path, composer will look in this path before trying to fetch the packages from https://packagist.org when you run the composer require command. However, you can also disable the packagist.org repository completely if you want.

Now where to get the actual composer packages? If you have recently installed any package, you can find them in the composer cache directory which is usually at C:\Users\<username>\AppData\Local\Composer\files. Otherwise, you can run composer require foo command so that the file is saved there and you can save it to your storage. In theory, simply packaging the parent directory of the repo source containing composer.json file should also work but I haven’t tried that.

But remember that while installing, you must ensure to install dependencies first. For example, if foo package depends on bar, you must install bar first.

[2] The “copy package” way

This one is a bit tricky and hackish way and only recommended for experienced coders. Sometimes, you will want to just “copy” the package directory from one project to another. As the linked post suggests, you can just edit your app’s composer.json, go to the autoload section and add the mapping to this newly copied package as follows:

  "autoload": {
    "psr-4": {
      "App\\": "src/",
      "Funtastic\\FooBar\\": "lib/foobar/src"
    }

After that, you can run composer dump-autoload so that composer will integrate this package and it becomes available in your app.

Thoughts on PHP routing strategies

2024-05-29T00:00:00+00:00

Routing is a tricky business in PHP simply because there is no one standard way of doing it. Starting from the most simple and benign (but inflexible) routing strategy of handling every incoming request in the index.php script to the highly flexible (but arcane and opinionated) routing libraries of Symfony, Laravel and the likes, there are all kinds of options in between and it’s up to you, as a developer, what routing strategy to use.

As a freelance programmer, while considering a routing strategy for the long term, it helps to remember two major deployment constraints on your production server:

Your client may want the PHP app to be installed to a specific subfolder like http://xyz.com/subfolder/ instead of the root domain or subdomain.
Your client may not give you full permission to override the .htaccess file, hence any strategy to rewrite all URL paths to index.php may not work.

Especially for the second point, the common or usual approach is to simply override your .htaccess like this:

DirectoryIndex index.php
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php/$1 [L]

This simple Apache rewrite rule will send all script requests to your index.php where your routing component can easily handle it with something like this:

$router.get("/", function(){
	echo "Hello, World";
});

But what if you’re not allowed to override the .htaccess file? I imagine many modern PHP routing libraries will fall apart if that happens! But some frameworks like CodeIgniter provide useful functions to handle this situation. The site_url() function is especially useful here. Using it, I can refer to a URL route irrespective of whether the request came by overriding the index.php or not. Your user may have typed xyz.com/index.php/foo/bar instead of just xyz.com/foo/bar but in both cases, the request will arrive at the foo controller’s bar method. And if you want to redirect to the foo/bar URL, you can simply do it by using site_url() function:

header('Location: '. site_url("foo/bar"));
exit;

Instead of hard-coding the URL or making any assumptions, this is a better approach as it will work irrespective of whether .htaccess was allowed to be overridden or not.

Similarly, for the first issue of using a subfolder, there is the base_url() function in CodeIgniter. You configure this one time variable initially and use it throughout your code like this:

<link rel="stylesheet" href="<?=base_url()?>static/css/app.css">

I don’t have to hard code any static path here and the web app becomes extremely portable. I’m sure other modern frameworks like Symfony and Laravel must be providing equivalent features and I’m yet to look at them. But overall, I find these OOP frameworks much too complex, which is almost like using nuclear energy to ignite a kerosene lamp! Maybe Big Tech and large IT firms who are into scaling might find them useful but for small freelancers and bloggers, I think the old school PHP frameworks still work the best. What do you think?

Working with multi-queries in CodeIgniter

2024-05-27T00:00:00+00:00

Running multi-queries (a bunch of text containing arbitrary DML/DDL statements) is highly unreliable and not an exact science in CodeIgniter or even PHP for that matter. The Internet is filled with posts like this, this, and this but you can’t depend on these solutions in most situations due to the difference between how each database driver handles it.

Most of the query() or exec() statements often don’t run at all due to the batch structure (begin and commit trans) not handled properly as per that driver’s liking. It may also happen that due to one error in a single SQL statement, the whole text is ignored and yet, there is no error prompt, the folks will think that multi-query executed successfully when, in fact, it didn’t. Hence, it is often so tempting to do something like this in CodeIgniter but doesn’t always work (especially with SQLITE databases):

$sql = file_get_contents(APPPATH . '/core/init.sql');
$this->db->query($sql);

The only approach that is guaranteed to work reliably here is to break your multi query into individual SQL chunks by splitting the semicolon and then run each individual statement like this:

$sqls = explode(';', $sql);
array_pop($sqls);
foreach($sqls as $statement){
	$statment = $statement . ";";
	//echo $statement;
	$this->db->query($statement);   
}

This will naturally rule out adding of any comments or whitespaces above or below the statements as you would in a script because that might cause an error. However, a simple and clean SQL script such as this one will work flawlessly:

drop table if exists settings;
drop table if exists prices;
drop table if exists quantities;

create table settings (
	id integer primary key,
	key varchar(2000),
	value varchar(2000)
);

create table update_status (
	id integer primary key,
	idx int,
	last_update datetime
);

create table prices (
	id integer primary key,
	price decimal(9,2),
	price_dt datetime
);

create table quantities (
	id integer primary key,
	idx integer,
	qty decimal(9,2),
	qty_dt datetime
);

Troubleshoot the 'Invalid Package' error on Android APK installation

2024-04-10T00:00:00+00:00

When you try to manually install an Android APK on your phone, you are typically stuck with an error similar to this:

Stack Overflow is filled with posts like this, this and this one but apparently, there seems to be no consensus regarding the cause or even a generally accepted fix for this problem.

The very obvious thing to first do is clean the project and rebuild the APK in Android Studio. Many answers suggest this and if it works for you then well and good. But if rebuilding the APK doesn’t resolve this problem, there are three other solutions you can try based on my practical experience with Android development so far:

If you are signing the final APK with a key, it could be an issue depending on how you sign. Some older android versions may not support the newer V2 signature format, so it’s recommended to sign your APK using both V1 and V2 formats.
I found another solution from this post and it often works in many situations. What values you’ve set for minSdkVersion and targetSdkVersion properties (typically in the app’s build.gradle file) are often the culprit here due to the way android works. Apparently, some newer phones don’t like APKs which support versions lower than M. In my case, I had kept minSdkVersion to 19 (KitKat) and once I changed it to 23 (Marshmallow), this package error mysteriously disappeared! If your user base consists of older android versions, you’ll have to release multiple APKs to cater to that segment in this case.
Apparently, another way this error goes away is by setting the exported attribute for your main Activity in the AndroidManifest.xml like below:

<activity android:name=".LoginActivity"
	android:exported="true"
	>
	<intent-filter>
		<action android:name="android.intent.action.MAIN" />
		<category android:name="android.intent.category.LAUNCHER" />
	</intent-filter>

</activity>

Please let me know whether this resolves your APK package error through comments. Also let me know if you know some other way to fix this error.

Intro to DOMPDF - light and simple PHP library to generate PDF documents

2024-04-05T00:00:00+00:00

Generating PDF documents from your HTML output is a very common requirement and there are several open source libraries to accomplish this in PHP. I came across this need for my project recently and I evaluated many popular ones such as TCPDF, mpdf, FPDF, etc. But the one that truly stood up to my evaluation in terms of efficiency (minimal footprint) and ease of implementation was DOMPDF.

DOMPDF is a pure PHP component which can be seamlessly installed using composer:

composer require dompdf

This will pull the dompdf component in your vendor folder, it occupies just about 7 MB of disk space which is great for the kind of work it does. Once you add this component with composer, you can start generating PDFs right away using the Dompdf\Dompdf object:

$url = "https://example.com/";

$options = new Dompdf\Options();
$options->set('isRemoteEnabled', true);
$options->set('isHtml5ParserEnabled', true);

$dompdf = new Dompdf\Dompdf($options);
//$dompdf->setBasePath($url);
$dompdf->loadHtml(file_get_contents($url));
$dompdf->setPaper('a4', 'portrait');
//$dompdf->setPaper( array(0,0,822,848), 'portrait' );
$dompdf->render();
$output = $dompdf->output();
file_put_contents("output.pdf", $output);
//$dompdf->stream();
echo "Done";

Needless to say, an important caveat or limitation here is that like most other libraries, dompdf will not support any and every latest feature in the CSS specification. It is mostly a CSS 2.1 compliant library and supports a few CSS 3 properties. In practice, I’ve seen most simple HTML tables and paragraphs set with attributes like bg/fg colors, fonts, margins, etc. work flawlessly. Styling with Bootstrap 3.x works too but 4 and above still has some issues.

In the above example, I’ve taken the HTML output of example.com which has very basic styling to generate the PDF. Here is the generated PDF output.

In this code, we first set the options for PDF generation using Dompdf\Options object. I’ve enabled isRemoteEnabled as my HTML content uses external CSS links. I’ve also enabled isHtml5ParserEnabled as I often need some HTML5 features. You can find more details on their wiki page about these options.

After that, you create a Dompdf\Dompdf object and load the HTML.

$dompdf->loadHtml(file_get_contents($url));

After that, you set any other options you need (such as paper size) and call the $dompdf->render() method followed by $dompdf->output() method. Lo and behold, your PDF document is now generated!

You can either save the generated result using file_put_contents() as I’ve done in this example or you can also optionally call the $dompdf->stream() in which case the PDF will be output to the browser window directly.

That’s it, so simple! Enjoy your pdf generation process using dompdf and if you face any issues, let me know using the discussion links. Happy Coding.

We are not creating the "right" kind of AI

2023-11-30T00:00:00+00:00

Arts, crafts, literature, even things like computer programming have been the domain of humans since time immemorial. Machines were historically meant to do menial tasks where the human couldn’t afford the time or resources. The classic example here is the type-writer, a remarkable machine that helped humans draft letters in a standardized and mechanized form which could then be used for documenting information in an objective form.

The printing press was an extension of that, helped humans create more durable and finished works, or books out of the drafts which they had penned down, either type written or hand written. A plethora of other discoveries and inventions followed including the telegraph, telephone, gramophone, musical instruments, etc. right until the dawn of twenty first century when personal computing with advanced software became a very popular thing almost everywhere. All of them were utilitarian inventions, in the sense that they served some purpose in advancing and enhancing the tribe of humans. Even mobiles and smart phones are extremely utilitarian (though they also come with a lot of cruft!)

But AI is no such thing, there is nothing utilitarian at least in the current forms of AI which are highly discussed and talked about today. The only utility or purpose they seem to have is to get rid of workers and staff through automation, and this itself is often celebrated in social media as if an end in itself. But show me where is the utility in this, for the society, for the tribe?

In that sense, the machine (AI) is an ideal servant but not such a good master. For starters, it’s a clumsy concept to begin with considering you’re coercing a binary logic machine to do tasks which often have very little to do with logic - crafts like singing, writing, teaching, etc. aren’t pure logic things, there is also a lot of nuance, chaos and heart involved.

Another thing is that trying to take away creative jobs from humans and giving it to robots while the menial ones (like data entry, manual labor, household work, etc.) are still being done by humans is an insult to humanity itself. It’s as if the authoritarians want to tell the humans, “This is your worth at most, nothing more. You’re too lowly to do something creative or different or unique”.

Human skills, inquisitiveness and thirst for knowledge and craft will eventually die off if all creative tasks are taken over by the machines. Consider the already degraded thirst for GK or General Knowledge ever since google search became a thing. There was a time once when kids used to watch the KBC Show (Indian adaptation of Who wants to be a Millionaire) with tremendous awe and enthusiasm, but no longer today. Today, a simple google search will tell them the correct answer for the quiz question, why bother with discussing the actual answer, the lore and the facts behind it, etc! This is just one example of a creative skill which is rapidly degrading among humans and already taken over by Turing machines.

Imagine a world where everything is “AI generated”, everything including books, songs, music, movies, etc. One can imagine the irony in humans consuming and enjoying such content - the very same humans who have the potential to create poetry, prose, songs, software and even artificial intelligence itself!

I abhor to imagine the state of such a dystopian society which is so divided between the chosen few “haves” administering the AI, and the masses of “have nots” doing all the menial jobs just to sustain the “AI ecosystem”. Are we already seeing signs of such a dystopian society being created in front of our eyes? You answer.

Unpopular opinion: desktop GUI is the most efficient and fulfilling way of human-computer interaction

2023-11-17T00:00:00+00:00

In many ways, the classical paradigm of the “WEB” as a mere portal or information resource was the most likable and valuable one in the utilitarian sense. These were the earliest days of the Internet when most of the actual Human-Computer interaction took place in the form of a Windows Desktop GUI and people went online to just “look up” some information they wanted.

Then something changed in early 2000s as more and more of these sites started becoming “apps” and started doing the actual computing. It solved a real problem though as you no longer had to install these individual EXE/DLL/OCX files on every user’s machine and that ONE centralized place called “Web Server” was all that’s needed to be updated and maintained. Thus, some of the “fat clients” started offloading their work to the “thin clients” such as web-browsers more and more, and that was also a good place to be in.

But the worst thing to happen from an end-user utilitarian perspective was perhaps the “Cloud computing” paradigm. At this point, the Gods of Internet decided that even the “thin clients” no longer need to exist and that ONE client called the “Web Browser” should take on all functions of the operating system as per the directions received from that ONE place called “Web Server”. Circa 2008-10 is where I put the start of this kind of development, almost a decade or so ago from now.

That’s the trajectory we are roughly in right now as browsers, on their way to become a fully fledged OS, are getting stuffed with updates upon updates every week and shoved down the user’s throats. This approach has created several problems for everyone:

Too much bloat as one software must cater to everyone’s needs. The “least common user” problem had always existed, no doubt, even with real OSs like Windows and even Office Software. But an OS has to just do the bare minimum of bringing up basic services like graphics, sound, background services, etc, they don’t have to worry about business logic or application layer. Web browsers trying to become OS, on the other hand, do have to worry about ensuring that each and every site loads correctly based on W3C standards and sadly, these standards are defined by what’s good for the enterprise sector than the ordinary or average end user.
Open invitation to authoritarian software like DRMs, Adware, blobs, etc. DRMs and Adware have already made their entry into most browsers and though there are extensions like ublock-origin to block adware, there is also a sustained effort to fight them with all kinds of things like restricting their features through the upgraded manifest version itself, and things like coercing the user to disable those extensions in order to view the content.
Use of Internet as propaganda tool. Because capitalism is very much built into our system, what is stopping those with lot’s of monies from using it to push propaganda through these ads or even just push propaganda in pure form? Users become sitting ducks to the subtle propaganda perhaps even without realizing it.
Inefficient and bloated JavaScript. Instead of taking time-tested and efficient best practices of desktop development in C#/C++/Java and putting them in the browser, what they’re doing now is use the bloated JS/NPM paradigm to write even desktop apps using frameworks like Electron! In this paradigm, the desktop app is a whole bundle of universe containing the Node.js web server, the Blink rendering engine and a browser like chromium that renders the UX. There are many ways to achieve platform independence across different desktop kinds viz. Windows, Linux and Mac, but using a layered solution like Electron is the worst way to go about it.

I don’t know where the future of web is going, how long this “cloud and data” related madness will continue but until it lasts, those who can see the problem here can revert back to using desktop apps as much as possible. Desktop apps are typically seen as intrusive and distrustful these days, thanks to clever marketing by the cloud folks, but more and more desktop apps are getting open sourced every day and if you think about it, an open source desktop app can be way more trustworthy than a website containing blobs of minified JavaScript who’s working nobody knows?

Let’s start celebrating desktop apps like we used to earlier. They’re not “old” as is often said, but they are classical. It’s the Internet that ages by leaps and bounds, not the real world and desktop apps are as close to that real world as one could get!

Managing drafts in Jekyll while writing blog posts

2023-11-12T00:00:00+00:00

The conventional wisdom of maintaining draft posts in Jekyll is to store them in _drafts directory.

Drafts are posts without a date in the filename. They’re posts you’re still working on and don’t want to publish yet. To get up and running with drafts, create a _drafts folder in your site’s root and create your first draft:

├── _drafts
│   └── a-draft-post.md
...

But I found this conventional approach a bit cumbersome to follow as a daily driver. For one, at each point of time when you decide that a draft such as a-draft-post.md is done version, you must copy it from _drafts to _posts and also rename it with the date part.

One solution here is to write a one time bash script or something for your workflow which does that automatically when run, but if you don’t want to do that, there is a cleaner approach which I’ve found:

The other way of declaring drafts in Jekyll is to save it in the _posts itself as usual, and simply set the published attribute in the front-matter section to false:

---
layout: post
title: '[Jekyll] Maintaining drafts while writing blog posts'
tags: jekyll tricks
published: false
---

Saved as _posts/2023-11-12-jekyll-maintaining-drafts-while-writing-blog-posts.md

This way, all you have to do is set the published attribute to true and push the changes to your github repository! Of course, the downside here is that you might forget to change the date at the time of doing that and your published post will end up being back-dated by a couple of days. Personally, I’m quite tolerant of that and the difference won’t be much as I don’t like to keep a post in “draft” status for a long time anyway.

In any case, this is just one approach for handling draft posts in the Jekyll workflow, there could be others.

How to allow Ping (ICMP) requests through the Windows Firewall

2023-11-09T00:00:00+00:00

The ping.exe is a great tool provided by the Windows operating system to check and ensure if we are connected to another computer located across the Internet or even our own local LAN through the ICMP (Internet Control Message Protocol).

However, it often happens that even if the remote machine is online, the ping request fails due to firewall or other settings and throws a “General Failure” or similar error message:

If your Internet is working fine otherwise and you’re able to browse the websites, the culprit in most cases is a Windows firewall rule which is blocking the outbound ping requests. So, head over to the Windows firewall settings through the Control Panel->Administrative Tools or by simply running the wf.msc tool in the run dialog. Open the outbound rules settings and you should be able to see the following three rules set for ICMPv4-Out echo requests as shown below. Make sure the “File and Printer Sharing (Echo Request - ICMPv4-Out)” rule is enabled for the currently active profile (Domain/Private/Public).

Now, unless you’ve changed something drastically in these firewall settings, there is a good chance that that they should be enabled by default. If not, then simply enable them and your ping requests should start working after that:

Now, it might happen in some cases that enabling this firewall rule for ICMP still doesn’t work and you’re still getting the ping errors. This depends on whether the currently active profile for your network connection is Domain, Private or Public, and which remote machine you’re trying to access here. If you observe the firewall rules closely, you’ll see that only the Domain profile is allowed to make a ping request to remote websites such as google.com, other profiles are limited to ping only in the local subnet:

Microsoft, in their great wisdom, thought that if your connection profile is Private or Public, you shouldn’t have any need to ping an Internet resource like google.com! In this case, you’ll have to either change your network connection profile to Domain through the control panel, or simply add a new outbound firewall rule that allows all outbound ICMP requests. If you choose to do the latter, you can then disable the built-in rules that you enabled earlier.

Hopefully, your outbound ping requests should start working by now!

How I made my web pages load 10x faster

2023-06-07T00:00:00+00:00

Update: As of 17th June, 2024, I have also started using font-awesome and google-fonts which goes directly against my principle of “no-cruft”! And while the point of this article still stands in theory, I came to learn that there are pragmatic limits to how far you can stretch minimalism. At some point, utilitarianism starts overriding minimalism.

Update: As of 1st November, 2023, I have switched to the default Jekyll site generator from Pelican as I found the deployment and Github Pages workflow easier. However, the point of this article still stands.

The most typical advice you get when it comes to improving site performance these days is purely technical things like using CDNs or other serving optimization, using X hosting mechanism instead of Y, going serverless, cache optimization, etc.

I’m not talking about those things, I’m talking about the old school millennial way of optimizing things which is by reducing clutter and subscribing to the ideals of minimalism and common sense!

The way our websites look and perform reflects our mindset in some ways, all that clutter and clunkiness that could be easily avoided still makes it to the web design for God knows what reason and keeps clogging our network bandwidth.

“It’s too bland and simple Prahlad, it doesn’t look professional enough”, they keep saying. Unfortunately, most of us have been trained to appreciate complex and clunky solutions in life instead of simple ones! This simple and bland looking eBay website from late 1990s, for example, allowed you to do a bunch of useful things with way more efficiency than today’s equivalents. From allowing a user to register and login to adding items to a shopping cart to processing the payment by directing the user to a gateway and then bringing them back to the “thank you” page - more utility, less clunkiness.

Today’s apps also do the exact same thing but they require 10x more resources due to all the bloat and clunkiness across the development stack. We need a hundred different npm packages for transpilers, minifiers, compressors, etc. and still the end result is clunkiness in user experience. The websites of 1990s used to run on a server with 32 megabyte RAM chip and powered by an Intel 486 processor!

An attitude of simplicity and minimalism is the urgent need of the hour in both our society and IT systems. Once you start questioning your confused and complicated stack with, “Do I really need this X layer or Y framework or Z library?”, you’ll find that the answer you’ll get is “NO” in most of the cases.

A few years back when this minimalism thing stuck me, I decided that I will de-clutter my website as much as possible. The first thing I did was to get rid of the free shared hosting and migrated my site from a wordpress.org instance to a statically hosted github pages site. Some time after that, somewhere around the pandemic period, I decided to get rid of my domain name too. Why waste INR 1,000 each year when github.io provides me a subdomain which is sufficient for almost every use case I can think of as a freelance programmer.

This is the kind of minimalist attitude we must bring in our lives. Imagine if a corporate IT or MNC firm decides to use its resources with such efficiency, imagine the kind of savings it’d make. All those savings can be used to pay off their engineers who are getting laid off and retain them instead of spending on costly AWS services which sustain that clunkiness!

If you think that even after switching from WP to a statically hosted site, was it even possible to bring even more minimalism to a website, then the answer is a resounding YES! Below are some measures which I took (and still taking as it’s a work in progress) for this:

Get rid of front-end frameworks like bootstrap and material-ui, write your own CSS/JS.

Whether it’s old school stacks like jquery/bootstrap or more modernish ones like react/material-ui, they all add some clutter and clunkiness to your site. People today have somehow got this toxic and pathological belief that each and every website or app must absolutely use these frameworks and libraries in order to look professional. I have nothing against these tools but it’s quite apparent that they aren’t needed for all use cases. A statically hosted blog such as this one is a perfect example.

If you are building a large ecommerce site or something, it could be justified but even there, you should ask yourself why you need a tool before just implementing (consider the eBay site from 1990s example).

Thus, bootstrap and jquery were the things I got rid of as I didn’t need them.

Use UNICODE emojis instead of icon libraries like font-awesome.

A clunky icon or image library doesn’t need to be downloaded each time if all you need is to display some symbols. The Unicode standard has evolved enough by now so that most symbols and emojis are part of the standard and available on all devices. As you can see on my about page, I’ve used emojis like books 📗, tools 🛠, pointing finger 👉 and even hour-glass ⌛ to convey my point across. You don’t need any icon or image library for this.

Resize/Compress your images and resources.

Do you even need that hi-resolution and thousands of pixels wide image which occupies triple digit MBs on the hard disk? In most cases, you don’t. In most cases, you specifically control the “width” or “height” attribute of <img> elements which usually need to be much smaller than their actual size. But the entire image is still downloaded to the client browser each time. Better idea is to just compress the image using a software like GIMP or Paintbrush by reducing its dimensions and that’s it.

I haven’t gotten around minifying my CSS and JS yet because the site’s average load is hardly ~500KB as it is. But guess I’d be able to shave off some more on that once I implement this in the tool chain.

Convert your PNGs and JPEGs to WEBPs.

This thing is still a work in progress but I’ve started converting my site’s PNGs and JPEGs to WEBP format. On average, WEBP lossless images are 26% smaller in size compared to PNG. These savings mean gigantic potential efficiency in terms of less bandwidth utilization and resource consumption over the lifetime of a website! It’s a wonder why we are still stuck using PNGs and JPEGs at all and the web hasn’t already moved to WEBPs entirely yet!

It’s all about minimalism attitude and mindset.

Think why is it that sites like Hacker News, Tildes, Lobsters, etc. are getting so popular among the nerds and intellectuals of this society. Think why is Old Reddit still up and running despite them trying everything under the sun to publicize the new and clunky version when it came! When Wikipedia tried the clunkiness way some time back, the Internet soon start filling up with help posts like these for switching back to the old Vector theme. The world needs to understand that minimalism is the future, there just aren’t enough resources in this world to sustain the clunkiness anymore.

Internet bandwidth also isn’t an abundant resource. It tended to be cheap ever since the data revolution began about a decade ago but that may not be the case always. ISPs across the world are already tightening the screws on this, they are gradually increasing the data cost. Jio and Airtel recharges are much pricier today than they were a year ago. Once that starts happening, your cloud services will need to bring more efficiency and performance instead of more features in order to stay profitable, better start preparing for that day today itself instead of waiting for that day!

And don’t fall for the wrongful narratives and agendas of those who keep justifying the clunkiness as part of their business model. Merchants of complexity will always keep building their Death Stars because it’s their job as a business or corporate to think about their interest. But at the same time, it’s our job to not get distracted by it and stay minimal which is both in our and society’s best interest.

In conclusion, this famous quote by MK Gandhi perhaps applies more in today’s world than it did in 1947:

The world has enough for everyone’s needs, but not everyone’s greed.

The power of minimalism: Why ecommerce and corporate websites need to de-clutter

2023-02-01T00:00:00+00:00

In the digital age, ecommerce and corporate websites often overwhelm users with cluttered designs and excessive elements. But as consumers gravitate toward simplicity, adopting a minimalist approach can not only improve user experience but also enhance site performance. Here’s why minimalism is the future of web design.

Why Minimalism Matters for Web Design

In the past, flashy designs and feature-packed websites were the norm. However, users today prefer clean, simple, and functional designs. Minimalism in web design focuses on eliminating unnecessary elements, creating a smoother and more enjoyable experience for your visitors.

The Cost of Cluttered Ecommerce Sites

Cluttered ecommerce sites can cause user frustration, slower loading times, and even a higher bounce rate. Here’s why reducing clutter is essential for any business website.

How to De-Clutter Your Website for Better Performance

Use whitespace strategically to guide users’ attention to important content.
Limit the number of colors and fonts to create a cohesive visual experience.
Remove unnecessary scripts and media files that slow down page load times.
Simplify navigation by reducing the number of categories and menu items.

Inspiration for Minimalist Web Design

Looking for inspiration? Below are some well-known brands which embraced minimalism and reaped the benefits. These examples demonstrate how minimalism can enhance user experience by focusing on simplicity, functionality, and clarity. Whether it’s a search engine, ecommerce platform, or content-driven blog, minimalist design helps eliminate distractions and guides users to the most important parts of the website.

Google Search Google’s homepage is the epitome of minimalism. It’s simple, functional, and focuses on one thing: search. The absence of clutter makes the user experience seamless, and users know exactly what to do when they arrive on the page. A single search bar, minimal text, and lots of whitespace is all they need!
Dropbox Dropbox’s website emphasizes simplicity, focusing on clear calls to action (CTAs) and minimal design elements. Their homepage uses a simple layout with large, easy-to-read fonts and a consistent color scheme. A Straightforward navigation, clean typography, and no unnecessary design elements is what Dropbox users are looking for.
Apple: Apple is known for its sleek, minimalist design philosophy across both its products and its website. The website is clean, with plenty of whitespace and a focus on product visuals. It uses large, high-quality images to showcase its devices without overwhelming users with unnecessary text or elements.
Medium: Medium’s design is centered on readability and content-focused layouts. The platform uses a minimalistic interface that puts content first, allowing users to focus on reading without distractions. Its design makes effective use of whitespace and simple typography, which contributes to a clean, clutter-free experience.
Airbnb: Airbnb balances minimalist design with high-quality visuals. The homepage features large, immersive images, yet the interface remains simple and user-friendly. Their focus on content clarity and ease of navigation ensures users can find what they need quickly. Minimalist Features like bold images, clean typography, and intuitive navigation is what makes Airbnb stand out.
Spotify: Spotify’s website leverages a minimalist interface, allowing its bold color choices and high-quality visuals to shine without overwhelming users. Navigation is simple, and calls to action are clear. Features like dark backgrounds with bright accents, minimal text, and prominent CTAs make Spotify website shine.
Zen Habits: Zen Habits is a blog centered around simplicity and mindfulness, and its design mirrors that philosophy. The site is purely text-based with minimal visual distractions, focusing entirely on content.
Everlane: As an ecommerce site, Everlane focuses on showcasing its products with high-quality images and minimalistic design. The clean interface allows users to browse effortlessly without being overwhelmed by too many elements.

Conclusion

Adopting a minimalist approach to your ecommerce or corporate website can improve user experience, reduce load times, and increase engagement. Start de-cluttering your site today and see how simplicity can lead to success!

"API first" apps are the future

2023-01-31T00:00:00+00:00

Apps tend to become very complex beasts, especially when they start to scale. Part of the complexity is due to the fact that the users could be a diverse set of people with multitude of tastes and preferences which could all be conflicting.

A classic example here is this recent tildes discussion thread I had started expressing my displeasure at the new Wikipedia design. As you can see from the discussion, it soon became quite heated with both for and against arguments with regard to the new design.

The nature of life is such that you can’t fit all people into just one mold. There are those who prefer to read wider paragraphs and others who like shorter ones. The trend seems to be moving towards the latter if the new design is indeed based on mass preference.

Many proponents of the new design will cite data and statistics to “prove” how shorter width is actually better for reading comprehension. But that’s not how life works, unfortunately. User preferences are based on their feelings. If you “feel” that the banana you’re eating is sweet then all the scientific data, research, etc. in the world stating that it’s bitter instead won’t change your mind.

Thankfully, Wikipedia has other classic or old school themes which they’ve retained just like old Reddit, so this problem is solved. There is also the case that Wikipedia is based on the open source PHP software called Mediawiki which can be accessible through an API also. Those having an issue with the “official” font-end can thus use the API to fetch the wiki articles and display them in whatever format they want.

The business wisdom here is that more and more Internet users are becoming “power users” now, they have strong preferences about UI and aesthetics, and they’re prepared to put some effort to improve that experience. The number of these power users is only going to increase on the Interwebs if this trend of the past continues.

As a commercial app developer or firm, you can gain an edge here by offering an “API first” platform. What this means is that there is a clear separation in code between the user interface and backend logic. The interface is just a shell or front which calls the “headless” APIs on the backend to do almost everything the app needs like logging into the system, fetching the data related to business logic like customers, orders, invoices, etc. and much more. Another advantage here is that you can improve your font-end code without disturbing the back-end logic which is coded in the APIs.

The user doesn’t need to care about how the data is actually fetched by the app. However, some of those users are going to be power users who may want a different interface or font-end features. They could be programmers too who can write a python script or something to suit their own taste provided an API exists. You, as an app developer, can cater to that need by bundling the APIs as a feature of your app and making the app itself “API first”.

Kiwixotherapy: A weird but working therapy for introverts suffering from sleeplessness

2023-01-14T00:00:00+00:00

If you’re an introvert who is suffering from both sleeplessness and also low self-esteem or impostor syndrome, I want to suggest you this therapy which I recently landed upon myself and seen many people recover from their insomnia or sleeplessness problem using it, and leading much happier and energized days now.

You’ll benefit from this therapy provided:

You’re an introvert who enjoys reading a lot.
Your insomnia is primarily due to Internet or social media addiction (continuously watching “unputdownable” shorts on Youtube, TikTok, etc. on phone).

The ideal advice here is to just put your phone somewhere far away from your reach and inaccessible so that the only activity you force yourself to do is do nothing but have a sound sleep.

But that’s not always possible. For one, the amount of toll it might take on your will-power could be enormous (there is a reason they’re called “addictions”!). Secondly, I’ve realized over the last few days that the reason most people always want to stay online is that at some deep level, they want to have a constant sense of connectedness with the world or universe at large. If this connection is broken, they’re staring into the abyss or void where the gravity of reality brings all the miseries of the world.

It’s almost like a red pill or blue pill thing, isn’t it! One way to gracefully come out of this illusory necessity you’ve created for yourself is to give your mind a substitute: An alternative “offline” version of that universal interconnectedness, one that doesn’t keep you awake (like youtube shorts) but one that’s reading material which will both enhance your knowledge and also gracefully make you fall asleep once you’re up for too long (like an encyclopedia article).

You see, a short video is just too interesting to watch and grabs your full attention to the point where it’s almost impossible to fall asleep (until that point of forceful no return like 4-5 AM which is really the time of waking up, not falling asleep!). An article or book, however, isn’t that attention grabbing. On the contrary, reading actually makes it much easier to fall asleep. How long do you recall you ever kept reading a book with the night lamp on without feeling drowsy and falling asleep right away?

And this is exactly where Kiwixotherapy comes in! Kiwix is an open source app which is specifically developed to enable you to download and read Wikipedia (the popular encyclopedia) entirely offline! The app is available for all major platforms including Windows, Mac and Android on their home page. Plus you can also get it from Google Play and F-Droid stores on mobile.

All you have to do is do a one time download of a Wikipedia dump through the app - the full version is about 95GB but smaller dumps are also available for most popular 50K articles, etc. which come in more manageable sizes of around 5-10GB. After that, you must make a habit of forcing yourself to disconnect from the Internet at bed time. If you feel like not disconnecting, remind yourself that there is a whole universe of knowledge and wisdom just waiting to be explored on your mobile phone! And real knowledge too, not just some opinions or trolling posts by self-proclaimed wizards on social media!

As you start reading the wikipedia articles while fighting your sleep, one of these two things will happen: either your knowledge will increase or you’ll fall asleep, and both these scenarios are in your interest or well-being. So, good luck and try this therapy today.

How to handle long-polling of XHR requests in PHP

2023-01-02T00:00:00+00:00

A common need you often face in PHP scripting these days is writing a long-polling endpoint for things like sending notifications or other kinds of responses back to the client. While using something dedicated like node, cometd or websocket makes ideal sense for such things, there are use-cases when you want to accomplish this in PHP itself and don’t want to add any extra dependencies or components to your project.

In this post, I’ll show you how to achieve this to a limited extent in PHP. But how scalable this solution will be will depend on the threading model and configuration of your web-server (apache/nginx). The basic problem with long-polling is that unlike short-polling, the client won’t exit immediately. The client keeps waiting for a response and this will inevitably block a thread on your web server. The same principle applies even when you go for other solutions like websocket, cometd, etc. but these libraries or components hide this “wait” implementation away from your code.

A notable exception here is node.js. Due to its uniquely architectured single-threaded event based model, this problem isn’t faced. Anyway, coming back to PHP implementation, here is how I handled it in my code. Please note that it’s extremely important to call session_write_close() as early as possible in your code and it’s also important to override the default max_execution_time php configuration at the beginning of your php script or framework configuration with something like this:

ini_set('max_execution_time', '300');

Once you ensure this, this is how you can handle it on php side:

<?php
public function fetch_long() {
	session_write_close();
	error_log('LONG POLLING');
	$start = time();
	error_log("LONG POLLING LOOP STARTS...");
	while((time()-$start) <= 300) {
		$sql = "select * from notifications where id=1";
		error_log("NOW QUERYING...");
		$r = $this->db->query($sql)->result_array();
		if (count($r)>0) {
			error_log("GOT NOTIFS, ESCAPING...");
			echo json_encode($r);
			return;
		}
		else {
			error_log("I KEEP POLLING ...".(time()-$start));
		}
		sleep(5);
	}
	error_log("ESCAPING ANYWAY...");
	echo json_encode([]);
}

Please note that once you call session_write_close(), you won’t be able to write to any session variables. However, you’ll still be able to read them. What I then do is create a blocking loop with a maximum limit of 300 seconds (you can increase it to as much as you want subject to what value you configured in max_execution_time earlier). If a notification is fetched before that time, I simply return it and exit the function. If not, the loop just times out in the end and I return a blank array.

Finally, this is how my AJAX call on front-end looks:

function fetchNotifications() {
	$.ajax({
		url: "/home/fetch_long",
		type: 'GET',
		cache: false,
		error: function() {
			setTimeout(fetchNotifications, 3000);
		},
		success: function(data) {
			handleNotif(data);
			setTimeout(fetchNotifications, 3000);
		}
	});
}

$(document).ready(function(){
	fetchNotifications();
});

Once the client receives a response from server, it keeps calling it again and again ad-infinitum to keep fetching the notifications.

The obvious limitation of this method is that a thread is blocked for every “waiting” request on the server side, so it isn’t ideal for scaling. In my case, I had only 30-40 users at most who used this app, so it wasn’t a big issue. Even then, I’ve added a pause of 3 seconds on client side before making the next request, just so as to give some breathing space to the server.

Despite the limitation, the app is working decently on the apache server without any configuration tweaks required. And besides, this is an architectural limitation which will always be there as I said, even in case of third party solutions like cometd and websocket.

Were you able to scale your PHP app with long polling by using this method? Please comment and let me know.

Common sense keyword research: the quickest way to find niche ideas for free

2022-12-08T00:00:00+00:00

Of late, I’ve found that a lot of unnecessary time and effort is being wasted on keyword-research activity by most new bloggers. Most bloggers do keyword research primarily for finding a niche i.e. they try to find a (relatively low-competition) topic related to their own where opportunity to rank is there but competition is less.

Inspired by many Youtubers, the usual way they go about this is by using the so called “keyword research tools” like semrush, ahrefs, etc, though there are some free ones too like Google’s keyword planner and ubersuggest. They then start bogging down into all kinds of number crunching and statistics like keyword volume, page and domain authority of competitive sites that rank for those words, then study their back links, etc.

In this article, I’m going to show you that none of this is really needed for ranking a good content and if you consider the overall picture, is either an absolute waste of time or even actively harmful in some cases!

To understand why this is a waste of time, let’s go back and understand the purpose of SEO in the first place. It may come as a surprise today but the dictum Content is King still holds true even in 2022. Eventually, Google is trying to rank content that caters to the query maker, not the individual keywords.

Keywords are just a means to an end, so bogging down to the levels of PA/DA/back links/etc. just defeats the purpose, be it for the blogger or the audience. If you’re confident that the content you’ve created is good enough (informative or provides a solution), the only kind of keyword research you’ll need is of the “common sense” kind which is, of course, entirely free.

By extension, this common sense keyword research is needed only to ensure that you’re writing about a topic which is in demand and your content won’t get lost in those unfathomable depths of Internet that don’t rank at all! For however good a content might be, it won’t rank unless there is demand for that content and that’s where the common sense approach to keyword-research comes in.

The common sense niche finding logic goes as follows. Taking my own case study as example, as a relatively new blogger and non-expert in SEO, it’d be a tad difficult for me to rank for something like “common sense keyword research”. Hence, I try to drill down or “niche down” my topic here by adding an “a” and putting the query into the Google search box. This gives me the following keyword suggestions:

Thus, you may continue postfixing “b”, “c”, “d”, etc. to your original or seed keyword to find more and more ideas which are in vogue presently. Google search suggestions is the common sense technique! It’s a simple and working design as Google is showing you these suggestions because people are actually searching for these words. Otherwise, Google would never have suggested them to you.

None of the other technicalities like search volume, page and domain authority, back links count, etc. should matter to you as blogger or content creator. Once you make a note of these keywords, it should give you a good idea of where the general trend is going related to your topic. And thus, incorporating more of them in your future blog posts should increase your chances of ranking in Google searches, it’s that easy and that simple!

How to handle multiple queries in SQLite using CodeIgniter

2022-11-14T00:00:00+00:00

My earlier article dealt with multi-query issues of MySQL, this one is dedicated to SQLITE. Multi-queries are often discouraged to begin with but there are times when you find yourself using them. One typical use case is populating the database initially when it’s empty. You do this by running an SQL script which may include multiple queries for creating tables, views, stored procedures, etc. and a few insert queries to populate default records (such as an admin user).

$sql = file_get_contents(APPPATH . 'core\\install.sql');
$this->db->query($sql);

The above is what you typically do when it comes to Codeigniter framework but when I tried this (using sqlite3 database driver), I was shocked to know that the multi-query didn’t work at at all. Only the first SQL statement (among a soup of them separated by semi-colon) was working.

So I googled it and came to know that it was an old bug. Apparently, they have now fixed it in later versions of PHP but only if you use sqlite_exec() instead of sqlite_query() function to run the query. Now, obviously our Codeigniter’s database abstraction layer was running the latter because I was still facing this same bug as of PHP 7.

So I tried to scavenge and find out what exact function Codeigniter was running for sqlite3 driver by studying the file system\database\drivers\sqlite3\sqlite3_driver.php. And here I found out that the driver was running the sqlite_exec() or sqlite_query() based on the very weirdest logic! Apparently, this driver tries to find out the first word in your whole soup of SQL and only if it starts with one of these (DDL) statements, it ran the sqlite_exec(), otherwise it ran the sqlite_query():

public function is_write_type($sql)
{
	return (bool) preg_match('/^\s*"?(SET|INSERT|UPDATE|DELETE|REPLACE|CREATE|DROP|TRUNCATE|LOAD|COPY|ALTER|RENAME|GRANT|REVOKE|LOCK|UNLOCK|REINDEX|MERGE)\s/i', $sql);
}

Now, my install.sql actually started with a bunch of comments explaining what the whole thing was about, I had to remove this comment block and ensure that it started with a CREATE statement, only then the multi-query worked with Codeigniter.

From userscript to Chrome extension: say goodbye to Greasemonkey and Tampermonkey

2022-10-14T00:00:00+00:00

Userscripts are a very useful and handy tool in the hands of power users. Just like customized garnishing, salt and pepper, etc. we put on our food before we eat, we can add custom javascript tweaks on the websites we visit.

For example, you may want Reddit to automatically highlight the recently posted (unread) comments on a post or thread. Another example is that when you perform a Google search, you may want trusted sites highlighted specifically (based on a pre-filtered list if you have one).

The classical or traditional way of running these userscripts is by using the “monkey” extensions, the most popular of them being Greasemonkey and Tampermonkey.

These extensions allow you to write scripts that execute when you visit the particular sites for them. While this is a great way of using userscripts, the better and more efficient way is to always convert them into browser extensions.

Extensions run natively in your browsers as you’re using one less execution layer. Besides, some extensions like Tampermonkey also update themselves too frequently (perhaps to keep up with the browser updates) and that might cause compatibility issues.

While comprehensive extension development is beyond the scope of this blog post, the conversion from a userscript to a basic chrome extension is quite straightforward. All you have to create is a folder for your extension and create two files named manifest.json and content.js. The content.js is where you’ll place the javascript code presently in your script file. Two important points to note here is that there should be no use of special GM_* functions and you must not use any return statement unless it’s inside a function.

Here is a very basic manifest.json example.

{
"manifest_version": 2,
"name": "Reddit Plus",
"description": "Highlight unread comments and mark them in blue.",
"version": "1.0",
"icons": {
	"16": "icon.png",
	"48": "icon.png",
	"128": "icon.png"
  },
"content_scripts":[
	{
	  "matches":[
		"https://*.reddit.com/*"
	  ],
	  "exclude_matches":[
		"https://*.reddit.com/test/*"
	  ],
	  "js": ["lib/jquery.min.js", "content.js"]
	}
  ],
"browser_action": {
	"default_icon": "icon.png",
	"default_popup": "popup.html"
  },
  
  "background": {
	  "scripts": ["background.js"],
	  "persistent": false
	},
"permissions": [
	"activeTab",
	"storage",
	"https://ajax.googleapis.com/"
  ]
}

My userscript also depends on jquery and instead of calling it through the CDN, I’ve included it as a separate script under “lib” folder as “lib/jquery.min.js”. Having an icon.png helps you distinguish the extension form others. The background.js is required as an empty placeholder. So, there are 5 files in total (the last one being optional):

manifest.json
content.js
icon.png
background.js
lib/jquery.min.js

Once you have this structure, just go to chrome://extensions on your browser and switch on the “Developer Mode”. This will allow you to install unpacked extensions. Just click on the “Load Unpacked” and browse your extension folder.

That’s it! You can now test your extension by visiting the site just like you did with userscripts and the monkey extensions. This was for chrome browser which I happen to use but extension development workflow for other browsers like firefox or edge shouldn’t be much different than this. Let me know how your userscript to extension conversion goes in the comments below.

How to set default date to current date in MySQL

2022-10-13T00:00:00+00:00

The most typical way people set default values to mysql date fields is by using the CURRENT_TIMESTAMP constant as follows:

create table ledger_entries (
	id int(11) not null auto_increment,
	entry_date datetime default CURRENT_TIMESTAMP
);

This will obviously work if you need both date and time parts in the value. But in most cases (like entry_date here), we only need the date part and would happily like to save that extra space. Even if you don’t want to save extra space, you may want to use the date data type for widget compatibility on the front-end . In few such situations, we prefer to use the date mysql data type instead of datetime.

The obvious constraint here is that you can’t then use CURRENT_TIMESTAMP as the default value to the newly added rows. Following isn’t allowed in MySQL as no pre-defined constant (except CURRENT_TIMESTAMP) is even permitted as a default value in the first place:

entry_date datetime default CURRENT_DATE

Now what will you do in such situations? As stated in some stack overflow posts like this, this and this one, a new (2018) MySQL version does allow a default date like this but I don’t recommended this as the practice won’t be compatible with universal MySQL installations or versions:

CREATE TABLE INVOICE(
	INVOICEDATE DATE DEFAULT (CURRENT_DATE)
)

Another way to handle this is to write a trigger. When new rows are inserted to the table, you may want to set the value to CURRENT_DATE. You may prefer this approach if you’re already writing a trigger but it’s a tad extra code and inefficiency to keep adding triggers just for this one need though.

I felt the best way is to handle this at the application level, that’s what I preferred to do when I came across this problem recently.

Happy coding and let me know your experience with mysql date/time values in comments below.

References:

How to execute multiple queries in MySQL using CodeIgniter

2022-10-12T00:00:00+00:00

Multi-queries are often discouraged with mysqli functions but there are times when you must use them. One obvious use case is initializing the database. One of the first things your app must do is determine if the database tables exist or not, and then run an initializing SQL script if they don’t. This script may include multiple queries for creating tables, views, stored procedures, etc. and a few insert queries to populate default records (such as an admin user).

Running multiple queries with mysqli isn’t an exact science though! The mysqli_multi_query function is technically the way to do it but there are some quirks you must be aware of when using it.

A major challenge here is error handling. The above function sends the individual queries (separated by semi-colons) to server one by one and stops executing the moment it faces an error in one of them. As stated in this stackoverflow answer, there is no official way to fetch errors in each and every one of your statements.

What you must do, in fact, is keep calling mysqli_next_result() again and again until each query result (or error) is fetched. This is how a proper implementation looks like (the code would probably be much shorter if it was implemented using some other technology like python or ADO.NET!):

<?php
$data = array('error' => []);
if ($this->input->method() == 'post') {
	$sql = file_get_contents(APPPATH . '../init.sql');
	$result = mysqli_multi_query($this->db->conn_id, $sql);
	if (!$result) $data['error'][] = db_error(); //handle error
	while(mysqli_more_results($this->db->conn_id)) {
		$result = mysqli_next_result($this->db->conn_id);
		 if (!$result) $data['error'][] = db_error(); //handle error
	}
	if (count($data['error'])==0) {
		$data['info'][] = 'The system is successfully installed!<br><a href="/auth/login">Click here</a> to login!';
		$data['installed'] = true;
	}
}

Another quirk to be aware of with mysqli_multi_query is that you must ALWAYS fetch the results by calling mysqli_next_result subsequently until mysqli_more_results() returns true. Not doing so may introduce some inadvertent bugs in your code when you later try to fetch records through mysqli_query.

Happy coding and let me know through comments below how your implementation goes!

How to approach and evaluate programming languages for a project

2022-09-21T00:00:00+00:00

While it’s a topic which has already invited hair splitting debates ad nauseam, it has also invited a lot of hype and there is a need to unclutter that hype and think about this topic objectively.

The Toolbox Approach

This has been the classic or traditional approach when programming used to be much simpler in the earlier days! “Right tool for the right job” used to be the motto once and you never thought of using a hammer when a chisel was needed. Around the early part of this century, languages began to consolidate and the result is that we only have a handful of highly popular ones used in the industry today and the rest are cast away into specialized domains.

The Paradigm Approach

The paradigm approach once was and still is popular today, in fact this is one way of thinking that contributes to the toolbox approach. Procedural languages like C and Rust are naturally built for system programming work or in situations where that last bit of computing performance is needed. The OOP paradigm in languages like C++ and C# helped build most of the desktop world citizens we use today such as operating systems, office suites, internet browsers, IDEs, etc. Python’s dynamic nature is similarly suited for data analysis work and PHP’s “all paradigms embracing” philosophy once made it a popular choice for web development. Same could be said about JavaScript to some extent, though it’s a language choice that’s more like thrust upon the web developers instead of being a natural choice! This is a highly subjective view, of course, all kinds of arguments can be made here depending on your own tastes and preferences.

The Ecosystem Approach

This is what I consider to be a modern approach of evaluating languages. Programming is no longer simple like it used to be once, today each major language has a comprehensive open source ecosystem built around it, and since these communities collaborate among themselves to not only build redistributable software packages but also help their users on coding forums, etc., they become a very important factor or consideration in taking the decision to use that language itself!

Consider the strange case of PHP, for instance. It’s almost treated like a pariah in many programming circles, you’ll find very few courses having PHP in their curriculum and not one major coding forum, blog or discussion site has a PHP topic included except Stack Overflow. But still the language is almost entirely powered by its community or ecosystem! It’s also a language that pioneered open source development and collaboration by involving contributors on massive scales with projects like Wordpress, Drupal, Joomla, Laravel, etc. which have all become ecosystems in their own right.

A technology startup which is already hard-pressed in today’s economically stressing times will obviously want to leverage the power of open source software and free tools, and that’s where the ecosystem comes into picture! How friendly is the community on discussion forums and how large is the package ecosystem are typically the top questions in the minds of technology startup entrepreneurs.

In this regard, Python, PHP and JavaScript are the major champion ecosystems and perhaps top the list of usage surveys, they’ve developed elaborate package management systems to serve their bases - PyPi, Composer and npm respectively. This trend has caught to some extent in the C# and Java world too with nuget and gradle/maven but theirs seem to cater to mostly the enterprise world, there doesn’t seem to be the same kind of buzz among the startups for these languages.

Summarily, these are the three major approaches for evaluating a programming language. There are others too, a major one being local availability of professionals skilled in a language. But in the age of remote working, I’m hoping that factor should become less important as time progresses. And there are “contrarian coders” too who will invest in learning long obsolete languages like COBOL and FORTRAN hoping that the skewed supply/demand ratio for these skills arising due to the need for maintaining legacy systems will ensure they get hired with handsome perks. Indeed, that’s an approach or way of looking at things too!

How to organize your research for blog posts in 2022: A comprehensive guide

2022-02-07T00:00:00+00:00

One of the most challenging aspects of blogging is how to perform research and more importantly, organizing your research. As a technology blogger, I face this problem too, so some days ago, I did some research on… guess what? Yep, on how to organize research for writing articles or blog posts!

A natural question that arises here is why should bloggers organize their research in the first place? There are many benefits to it, the most obvious is that because your writing reflects your thinking or thought process itself, getting into a regular habit of organizing your research work will then force your mind to organize your thought patterns too. This will, in the longer term, make you a better thinking and more intellectual person, and also help your creativity as your creative mind will now have a much larger and more organized arsenal of knowledge to come up with more ideas more quickly.

The bad news, though, is that organizing research isn’t an exact science, there is no one true way of doing it and every blogger evolves their own method over time. Having said that, as Katie Holmes describes in this article, “Librarians are known for a few things. Aside from loving cardigans and tea, we tend to be Amy Santiago levels of organized”. The article borrows some great ideas from librarians in this regard!

Before straightaway diving into online research, let’s determine what kind of research we are doing first. Have you already decided on the exact topic and/or headline of your next article and doing research just for gathering content? Or you don’t yet know the topic and you’re just doing some “general” research around a broader topic to then find out what exactly to blog about? The latter is called preliminary research and you should ideally do it prior to performing the actual “heavy” research.

In fact, preliminary research is something I happen to do almost always as that’s how I come up with ideas for my next blog post. Accordingly, you may organize your note taking process. What I personally do is I’ve created two folders called “research” and “drafts” for the preliminary and actual kinds of research respectively. Inside the research folder, I have various topical subfolders like tech, health, lifestyle, etc. in which I store notes in the form of plain text files. The drafts folder is for my actual writing projects, in this folder go one file for each article I’m in the process of writing and these are in markdown format. The end result may look something like this:

├───drafts
│   │   how-to-get-near-perfect-shave-in-less-than-5-rupees.md
│   │   how-to-understand-determinism-with-flowchart-analogy.md
│   │   how-to-maximize-happiness-with-dmu-philosophy.md
│   │   why-we-should-support-community-distros-like-mint.md
│	│
│	└───fiction
│
└───research
    ├───finance
    │       nifty.yaml
    │
    ├───health
    │       yoga.yaml
    │
    ├───horticulture
    │       organic-fertilizers.yaml
    │
    ├───lifestyle
    │       frugal-living.yaml
    │       yoga.yaml
    │
    ├───tech
    │       github.yaml
    │       linux.yaml
    │       microsoft.yaml
    │
    └───writing
            organization.yaml

The markdown files are initially filled with chunks and subheadings, all organized into a rough outline until I finally call the research done and then start organizing the actual content of the draft. This way, I gradually come up with an article’s content bit by bit, as if picking up parts of a big jigsaw puzzle and joining them one by one. Trust me, doing all of this might sound very “boring” or monotonous initially but it’ll be real fun once you start getting the hang of it!

Apart from this, you can also maintain something called Concept Maps which Katie also describes in her article. It’s a form of organizing your brainstorming process. You start with a topic say python. Then you might link various areas of applications to it, say web development, data analysis, machine learning, etc. Then you think of various frameworks and apps like flask, django, mezzanine, etc. So, you can have something like this:

Web Development
  ├───Flask
  ├───Django
  ├───Mezzanine
Machine Learning
  ├───NLTK  
  ├───TensorFlow
Data Analysis
  ├───Numpy
  ├───Pandas

You don’t need any specific app or software to maintain this, you can even maintain them in something called yaml files (*.yaml). These are plain text files with support for hierarchical text storage and can work wonders for tasks like this if you have a good text editor such as Notepad++ or Visual Studio Code.

As for the preliminary research text files, I create one for each major topic. For example, github, microsoft, windows, linux, ssh, etc. (under technology). In each file, I’ll add a link or citation (for the source) and few bullet points for each below it (this is what Katie calls “cue cards” or “annotations”). For example, in case of github:

[1]:
	+ Microsoft acquires Github Inc.
	+ Speculations arise as to future of source hosting facility.
	+ Competitor Gitlab pushes on advertising.
	+ Many developers thinking of migrating to Gitlab.

[1]: https://www.businessinsider.in/Microsoft-has-been-talking-about-buying-GitHub-a-startup-at-the-center-of-the-software-world-last-valued-at-2-billion/articleshow/64420905.cms

A simple organizational trick I follow here is that I append the bullet points in my own words with a plus sign (+) whereas a dash or minus sign indicates that the quote is lifted from the article as it is. So, “Microsoft acquires Github Inc.” above is my own note.

Organizing information collected from various sources in this manner forces you to think like a journalist and believe me, you’ll come up with much more creative, original and refreshing content as a blogger, once you start practicing this strategy! It’s not necessary that you put just bullet points, you can have direct quotes in quotation marks, brief annotations or notes in your own language, etc.

The most important thing is to write down ideas as they come to you. Say you’re reading this above source (the Business Insider article) and one blog post idea suddenly strikes you: Microsoft’s Github Acquisition - A Perspective . Then stop reading it and create a new markdown file for this article in the drafts folder, and start writing down things as they come to your mind. Ideally, you’ll also have the github file in the research folder open in a separate tab of your text editor to borrow sources and bullet points from there as and when needed. This is the most organic way of coming up with articles and you can never go wrong with this method.

Alternatively, the idea for the github article may arise sometime later. For example, after you have had a good night sleep and wake up with a fresh mind next day. But since you already have everything organized in your computer now, it’ll be much easier to just open up your preliminary research folder and skim through your notes and thus equipping yourself with a much better chance of coming up with a good quality article. If you hadn’t organized this, you’d have to go through your browser’s history, bookmarks, etc., and do a tremendous amount of head-scratching just to recall what good points you had read yesterday!

Apart from this, it’ll also help you to read about the three types of sources as described in this article by Smiti Nathan of the Habits of a Travelling Archeologist blog. As bloggers, we mostly base our research on tertiary sources (like online news blogs, articles, Wikipedia, etc.) but it’s also important to be aware of primary and secondary sources in your field too if you want to establish some reputation as expert in your subject matter.

The article also describes various ways of finding sources like getting to know your nearby librarians, Google Scholar, locating literature reviews, checking out Academia or ResearchGate, engaging with live sources (experts in various fields), etc.

More inspiration for organizing writing content can be found in Mike Hanski’s article titled 5 Writing Tips: How to Organize your Research. Hanski starts by describing the importance of quality research which is highly lacking in today’s blogosphere and then proceeds to give some important tips like Record Everything, Always Be Ready for Genius to Strike, etc.

In conclusion, I feel that organizing research isn’t easy or straightforward, but if you’re motivated enough and prepared to dedicate it some time, you can do it successfully and evolve your own organization process over time.

Why I think "Sponsor Only" repositories introduced by Github is a terrible idea

2022-02-03T00:00:00+00:00

Today morning, I woke up to this dull news that Github is soon going to introduce “Sponsor Only” feature, which means the ability for an open source project to hide its source code from everyone except the one who sponsors them.

This incident shouldn’t be taken in isolation but as part of a larger trend. Few years ago, Medium also did a similar thing by allowing authors to monetize their articles. Quora followed suit too last year by prodding popular authors to sign up for their monetization program that involves putting their answers behind a walled garden.

I don’t even know where to begin on this. As I replied to the TechCrunch news tweet, I don’t like this general direction the Internet is moving towards. There used to be a time when open source used to stand for something, it was all about the commons or the community. Commercial technology companies (and even society at large) used to look up to open source as a model of humanistic value to cherish and endeavor.

This move is totally against the spirit of GPL/MIT/Apache licensed software which is all about openness and software freedom. In fact, both Stallmanists (FSF) and Open Source (OSI) camps are largely affected by this, so both should speak out vehemently against this. This is the time to forget your differences and unite against a common malady. Your definition of “freedom” and what constitutes it may differ, you both may have different ideologies, but here the very foundation of software freedom is being threatened with this “Sponsor Only” move.

Now, I understand that putting food on the table should be a developer’s primary concern, and there is nothing wrong in monetizing their efforts. But I strongly disagree with the “walled garden” approach many are taking for it, the same could be done by using more ethical means like advertising, affiliate marketing, repaying the sponsor in other terms (like putting out a brief marketing video for them on youtube as it’s usually done).

Meanwhile, I don’t understand the very confusing and convoluted concept of “sponsor” here who wants the product all for himself! The wordnet dictionary defines sponsor as “Someone who supports or champions something”. If one wants to sponsor a FOSS project, it’s naturally assumed that they’re interested in the open source or freedom friendly nature of that project too. If not, they should just stop doing the farce of this “sponsorship” and be direct in stating that they want to acquire or buy the project (for that is what they’re effectively doing with this “Sponsor Only” move).

I can understand an individual developer’s or author’s need for money and the need to put bread on their family table. But an institution like Quora or Github should rise above these things, they’re the custodians of the legacy of today’s Internet and all the content on it. Humans are fallible and corruptible, and that’s exactly why we have institutions, even commercial ones have a code of ethics or conduct apparently. Quora could have easily thought of better ways of paying off their content creators than coming up with a program that lures them with money in return to putting their posts behind a paywall. What kind of nonsensical approach is this? Everyone starts somewhere in this world, and everyone isn’t born with a silver spoon or the ability to pay for costly subscriptions. The pandemic has broken the financial backbone of many individuals as it is. This is the time to bring out the best of humanity inside us, not the worst.

Imagine a world tomorrow where every single website or blog on the Internet is put behind a paywall, a world where money is the sole motivation to create content and the size of people’s pockets determines what gets published. Do you like to live in such a world? Free thinking individuals who don’t like to let that happen should speak out in loud voices against these moves, otherwise they’ll only keep getting worse and worse.

I also understand that Capitalism vs Socialism is a never ending debate and there are all shades of grey on both sides. But I have a strong feeling that some endeavors like Medicine, Education and Open Source should exist in a realm where not money, but work ethics and human values are a motivator for doing things. Otherwise the result could be disastrous for the society as a whole.

Unlocking Creativity: Your Ultimate Guide to Overcoming Writer's Block

2022-01-24T00:00:00+00:00

Writer’s block is a common challenge that even the most seasoned writers face from time to time. Finding effective strategies to overcome it can feel overwhelming, but understanding the root causes is the first step. Drawing from various techniques, including the 10 content-types method, I’ve compiled a comprehensive checklist to help diagnose and conquer your writer’s block.

Writer’s block can feel like a “wholesome evil.” Its elusive nature makes it difficult to pinpoint the exact cause. Are you struggling due to isolation during a lockdown, or is an underlying issue like depression stifling your creativity? This mystery often leads to diverse perspectives, as highlighted by my recent post in the Creative Writing Facebook group, where writers shared their unique insights. Here’s a checklist to help you identify the reasons behind your block, offering potential solutions to reignite your writing flow.

1. Procrastination

Procrastination ranks high on the list of writer’s block causes. Many writers acknowledge it as a significant hurdle. This avoidance can stem from various factors, including perfectionism or anxiety. Recognizing this pattern is essential to overcoming it.

2. Lack of Creative Ideas

Sometimes, a dearth of fresh ideas can stall your writing. For non-fiction, researching your subject matter may help, while fiction writers might find inspiration through new experiences or breaking their daily routines.

3. Imposter Syndrome

This phenomenon is particularly prevalent among young professionals, manifesting as an overwhelming need for perfection. It can lead to self-doubt and the belief that you don’t deserve your achievements. Acknowledge your accomplishments and remember that even the best writers start somewhere.

4. Need for Perfection

Perfectionism often hinders progress in creative endeavors. As writers, we can get stuck trying to perfect every sentence. Remind yourself that it’s okay to create a draft that’s not perfect. You can always edit later!

5. Busy Schedule

Writers sometimes overlook the importance of dedicating time to the writing process. If you’re juggling numerous responsibilities, carving out time for writing can feel daunting. Reflect on your schedule and prioritize your writing time.

6. Distractions

External distractions can derail your focus. Whether it’s loud noises or digital notifications, finding a quiet space or using tools like earplugs can help you regain concentration. Practicing mindfulness meditation can also enhance your focus.

7. Depression or ADHD

Depression can be a significant barrier to writing, yet it often goes unacknowledged. Recognizing its presence is vital to addressing it. Consider seeking professional help if you suspect this might be a factor.

8. Lack of Confidence

Low self-confidence can stem from various issues, including depression and imposter syndrome. Reflect on your feelings and consider reaching out to fellow writers for support.

9. Ditz Factor/Cluttered Thoughts

A disorganized mind can inhibit creativity. Take time to brainstorm, outline your ideas, and organize your thoughts. Tools like Google Trends and Answer The Public can be helpful.

10. Need to Think “Outside the Box”

Creative thinking is essential for generating original ideas. Revisit this checklist regularly to stimulate your thought process and find what inspires you.

11. Lack of Passion

Consider whether you’re writing about a topic that truly interests you. If not, exploring new subjects may reignite your enthusiasm for writing.

12. Lack of Enthusiasm/Energy

Sometimes, even when passionate about a subject, you might lack the energy to write. Identify the source of your fatigue and consider practices that boost your energy levels, such as exercise or short breaks.

13. Lack of Discipline

Creative pursuits thrive on structure. Establish a routine that encourages regular writing, helping you push through periods of stagnation.

14. Creative Exhaustion

Overextending yourself can lead to burnout. If you feel creatively drained, take a step back, recharge, and diversify your writing topics to renew your inspiration.

15. Failure to Walk

Taking a simple walk can refresh your mind and spark new ideas. Don’t underestimate the power of movement!

16. Other Mysterious Forces

If you’ve explored all the above factors and still face writer’s block, there might be other reasons at play. Experiment with different approaches to discover what works for you.

By addressing these various factors and implementing targeted strategies, you can break free from writer’s block and unleash your creativity. Happy writing!

How to generate a never ending supply of blog posts

2022-01-15T00:00:00+00:00

If you’re suffering from writer’s block or don’t know what to write about or how to research content for your next blog post, this guide will help you. There are just about 10 basic content-types or patterns around which weblog articles are written these days, and by creatively combining them with your subject matter expertise or your knowledge of hobbies, crafts or something else, you can potentially end up with a never ending or infinite supply of blog posts throughout the year!

The 10 basic content-types.
Your Interests, Hobbies and Subject Matter Expertise.
Combine the 1 and 2!
Static and Dynamic Post Ideas.

If this sounds very simple to you, that’s because it really is. Writing isn’t a rocket science, all you need is a little organization and you need to keep some time aside for it, and you need to keep practicing it until you gain perfection like any other art. Once you start getting into the “zone”, writing will feel as natural to you as photosynthesis!

The 10 content-types

Listicle: A List formatted article or listicle is one of the most commonly found these days. The ones titled like “40+ Ways of Doing Something” fall in this category. These articles are relatively easy to write because you’re typically only playing the role of “information organizer” here. It’s totally fine if you describe each item in the list only briefly and provide links to other resources like Wikipedia, etc. where subject matter experts deal with that item in much depth or detail. Checklists and Cheat-Sheets are also types of a Listicle.
How-To Guides: A How-To guide (such as this one!) is also very common because these are quite informational and Googlers are always hunting for these! Blogging is still a relatively new and inexact science in the digital world, even the best SEO and Internet Marketing companies are always struggling for eyeballs by constantly improving content quality, gathering street cred on social media, creating backlinks, etc. While these techniques seem to work to a limited extent, it’s not an exact science and there is no guaranteeing that a given blog post is going to bring eyeballs for lifetime once written (despite all the SEO/SEM tactics).

Gone past are those days when people actually used to read the “user manuals” that came with their products, these days most of them simply Google that information and typically land up on a “How-To” article that describes that product. And it could well be yours if you’ve dealt with the subject thoroughly and it happens to be an in-demand topic or niche.
Infographics: Infographics is the modern way of enhancing the content quality and improving the information value of an article. If you’re creative with graphics software like Inkscape or Gimp, you can design infographics which is nothing but a summary of what you want to say in picture format with words creatively sprinkled here and there!
Comparison - X vs Y: Comparisons are also as commonly looked up by Googlers as How-To Guides! I myself remember the countless times I’ve run searches like “PHP vs Python”, “Wordpress vs Drupal”, “Django vs Flask”, “WinForms vs WPF”, “MySQL vs PGSQL”, etc. You may be a programmer but the informational article you’re writing doesn’t necessarily have to be about programming. You may well decide to write a technical article pertaining to your subject matter too. But you may as well compare just anything you may come across or want to compare, even Apples vs Oranges or Democrats vs Republicans!
Ultimate Review: Review posts are also highly sought for these days for almost same reason as Comparison posts. Was a new product or service recently launched in your area of expertise? Or even a drastically changed new version of an existing product or software? If yes, write a post about it and it’ll be in high demand because a new product typically arrives with lots of bugs, issues, doubts, new ways of doing existing things, etc. and users are always looking for a helping hand there.
Ultimate Guide: Ultimate guides are detailed write of a particular idea or process which people often find difficult or confusing because there is never one straightforward or easy way of doing things.
Case Study: These are the ones titled, XYZ Program: How to make 15K or More Each Month, or similar. It could be a Udemy course or an IT project or even an infrastructure project implemented somewhere or anything else of interest. People are always looking for case studies in order to study them and learn from them. Agile and Software Engineering is another area where constant improvement is needed, relatively new field, inexact science and all.
Personal Story: Most personal stories these days feature along the lines of How I moved from 1K to 10K in 18 Months but they don’t necessarily have to be. Making money online is just one of the hot topics out there and frankly, it’s highly over-saturated! In fact, you’ll have a far better chance of ranking and getting page-views in that little niche or expertise area you might be having than here. A story could be about anything, it could be about the struggles of your life in childhood, school and college days, etc.
Informational Story (Industry News, World Events): The blogger can occasionally wear the journalist’s shoes too by writing informational posts! Industry news relating your subject matter is something you’re most naturally equipped to write a lot about, but it doesn’t necessarily have to be. You can write an informational post about just any and every topic on earth from politics to weather. The dinner table conversations going on in your home (mostly about politics!) are an ideal thing you can convert into an informational article and publish on your blog.

Your Interests, Hobbies and Subject Matter Expertise

These can be almost anything but generally fall in the following broad categories:

Technology: Whether you’re a programmer, tester, QA/QC analyst or designer, you’ll definitely have lot’s of things to write and share about your field and even workplace, a weblog is an ideal platform to do this. It doesn’t even have to be about IT, you could even be an Auto Engineer or Aeronotics Engineer or someone else. Each person has a unique story to tell and so do you, and the Internet is always all ears for that!
Finance: If you work in the finance industry as an accountant or book-keeper or even a clerk, you’ll still have a lot to share about the changing dynamics in your field, the computerization and use of open source software, changing policies of the government and due to effects of COVID, etc. Personal finance, and tips on saving money are also great topics to write on.
Politics: Politics is almost like the bread and butter of netizens who read blogs, there are no other kind of articles that go viral in such a short time as those relating to politics! If you have a flair for writing creative and interesting matter on this topic, you’re definitely going to have a field day here!
Food: There are millions of food blogs out there already and yet, some new content turns up every single day! Some new delicious recipe or a variant of existing recipe or something different you tried out today? You can also blog a lot about topics relating to cuisine like health effects of various spices and vegetables, veganism and vegetarianism, etc.

Combine the 1 and 2!

Now, the real magic happens when you apply the basic content types to your knowledge or what you already happen to know! For example, do you shop a lot? You can then write a comparison post on say, Walmart vs Tesco or Big Bazaar vs Reliance Mart! If you’re a Python engineer, Flask vs Django is the ideal post in that case. And how about Veganism vs Vegetarianism! The possibilities of doing this are almost infinite!

Static and Dynamic Post Ideas.

The above process of creatively combining your knowledge with basic content types will give you what is called static posts. They’re called as such because the content is mostly static in nature (though their relevance will progressively get lower with time), and hence such posts can be planned in advance and organized using what is popularly called blogging calendar.

Now, because of how the Internet works, you’re only allowed to share a couple of posts each day on social media platforms at most, so you must plan your content accordingly. Writer’s muse typically doesn’t work evenly, they come and go in leaps and bounds. But you can’t afford to don’t interact at all for a week and then dump all articles at once on Twitter, that will most likely flag you as a spammer! What bloggers typically do is they write their drafts in advance (leaps and bounds creativity!) and then schedule to publish them evenly.

Apart from these static posts which you can plan in advance, you can also write a dynamic post which can be written and published any time. Some big event took place in your field all of a sudden, yesterday? You’ll have a lot to say about that topic, so you can come up with a new post and publish it immediately.

Your content strategy should involve publishing a mix of both static and dynamic posts frequently. Having only dynamic posts won’t give you much content at all because there will be only so much happening in your given field every day. On the other hand, having static posts only will make your blogging process feel more monotonous and less creative. So, having a mix of both is a good ideal.

Tackling a common CSS design problem: Footer placement at page or content bottom, whichever is lower

2021-04-25T00:00:00+00:00

This is one of those commonly occurring nags in web development which I’ve solved several times before but still have to scavenge the googles and stack-overflows each time I run into it. That’s why I’ve decided to document the simple solution to it in this brief article.

What happens is that if you position your footer div and fix it at bottom of the page (position:fixed, bottom:0, width:100%), it will work great on shorter content pages (where you don’t have to scroll). But the problem is that on longer pages too, instead of moving to the bottom of the content, it will be stuck there at the viewport bottom like an idiot!

The above situation can be seen in action in this fiddle where multiple “lorem ipsum” blocks (<p> elements) are placed to simulate content growth. You’ll find that the footer will work flawlessly when the content is short (only 1-2 “lorem ipsum” blocks) but the footer gets stuck at the viewport bottom as you keep adding the blocks and they extend beyond the viewport height!

On the other hand, instead of positioning your footer, if you just let it be (this is what about 90% of coders initially do), you have another problem. Your footer will now be placed correctly on longer content pages where you must scroll but on the shorter pages, they’ll be hanging in the middle of the page where your content ends as shown in this fiddle.

There could be multiple approaches to solve this problem. I personally prefer the old school method which is quite simple and easy to understand. Besides, it doesn’t require adding of any blank HTML element like “#offset” or “#placeholder” above your footer. All it requires is that all your HTML elements above the footer must be wrapped up inside one container div element. So, the body should be structured something like this:

HTML
..BODY
....div.container
......header1,
......article1,
........p,
......etc, etc.
....footer

Then all you have to do is set your div.container’s minimum height to viewport’s height minus footer’s height. Assuming your footer’s height is 55px, all you have to do is:

div.container {
	min-height: calc(100vh - 55px);
}

You can see a working demo of this in this fiddle. Even as you start adding more and more “lorem ipsum” paragraph elements, the footer will always be placed at the “right” place irrespective of other element’s positioning and content size! This is what you’d call a “properly placed footer”:

dateutil.relativedelta: A must have tool in your python kitty

2021-04-25T00:00:00+00:00

Making additions or subtractions to datetime variables is one of the most commonly encountered programming endeavors and that’s what the built-in datetime.datetime.timedelta object is for in python. Its very useful in adding hours or days to your datetime variables as follows:

import datetime

now = datetime.datetime.now()

tea_time = now + datetime.timedelta(hours=2)
next_week = now + datetime.timedelta(weeks=1)

But the datetime.timedelta objects have one serious limitation. You can add hours, days or even weeks to a datetime variable but not months! And there is a reason for that. As per the science of Horology (study/measurement of time), “months” isn’t a fixed quantity of time. There could be “long” or 31 day months like January/March, there could be 30 day months like April/June or even 28 day months like February.

However, apart from this theoretical concept of month, we also have a practical concept which we commonly rely on for our usual tasks. For example, if today is 25th of April, a month from now would be 25th of May. This practical concept just considers the difference between current date and the same day of next month (25th in this case) as one month while disregarding the fact that the actual duration of this difference may not exactly be a fixed quantity like 30 days. It could be 30, 31 or even 28/29 depending on which month you are counting from and whether its a leap year.

Computing this practical difference is also a very common programming endeavor but unfortunately, there is no easy way to do this in python using the built-in datetime package. But thankfully, the folks have invented this library called dateutil which does exactly that! Installing this package is pretty trivial and in most environments, you might be already having it as hundreds of popular python libraries rely on it.

pip install python-dateutil

After that, you can straightaway start using it to add or subtract months using the relativedelta object:

from dateutil.relativedelta import relativedelta

today = datetime.datetime.now().replace(hour=0,minute=0,second=0,microsecond=0)
epoch = today + relativedelta(months=6)

Of course, relativedelta is just one of the gems from the dateutil project, there are several others you can explore at https://dateutil.readthedocs.io/en/latest/.

Happy Coding!

The right way to serve static files when using django with gunicorn

2021-04-19T00:00:00+00:00

Yesterday, I learned during deployment that your Django app when used in combination with gunicorn will refuse to serve static files, do whatever you may. I looked up almost every Stack Overflow answer post on this topic including this, this and this.

I meddled with almost every hopeful setting including STATICFILES_DIRS[], STATIC_ROOT and STATIC_URL but to no avail. Its as if Django is designed to refuse serving of static files when using gunicorn and that’s what I started to suspect after everything failed.

And my suspicion was almost confirmed by this post which says that:

Gunicorn will only serve the dynamic content, i.e. the Django files

But I know that’s not strictly true because I’ve used Gunicorn with Flask in the past and it serves static files of your Flask app without any issues at all!

But then I thought that its better to handle static files using nginx anyway and since I was already using nginx as the front proxy on my server anyway, I thought of trying that post’s suggestion. As mentioned, I added a new location section to my nginx configuration file as follows:

location /static {
		autoindex on;
		alias /path/to/staticfiles;
	}

And that’s exactly what worked! I bypassed gunicorn entirely and static files are now being served directly by the front server and I think this is a more efficient setup than having gunicorn serve the static files.

But why gunicorn/django refuse to serve static files directly still remain a mystery. I think the problem lies somewhere in Django and not gunicorn because as I said, I’ve seen gunicorn serve Flask static files before.

How to enable twitter card validation for your blogspot blog

2021-01-18T00:00:00+00:00

When you publish a post on your blog, one of the first things you want to do is share it with the world, and Twitter/X is one of the most popular platforms for that. However, if you’re using Google’s Blogspot (also known as Blogger), your blog posts won’t automatically display as Twitter Cards when shared. Fortunately, you can easily configure your blog to show those eye-catching Twitter Cards by following a few simple steps.

In this guide, we’ll walk you through the process of enabling Twitter Cards for your Blogspot blog so your posts display correctly when shared.

Why Twitter Cards Matter

Twitter Cards allow you to enhance tweets that link to your content by including images, summaries, and other metadata. This makes your tweets stand out and improves engagement with your audience. By default, Blogspot doesn’t provide the necessary meta tags for Twitter Cards, but with a small tweak, you can enable them.

Follow these steps to ensure that your blog links expand into a Twitter Card when shared.

Steps to Enable Twitter Cards for Blogspot

1. Access Your Blog’s HTML Template

In your blogger settings, visit Theme > Down Arrow near Customize Button > Edit HTML in your Blogspot dashboard. This will open the HTML editor for your blog template.

2. Find the Post Section

Click on the HTML editor. Then press Ctrl+F to bring up the search bar, and search for the following tag:

<b:includable id='post' var='post'>

You may need to search more than once, as some templates contain multiple instances. We’re interested in the last occurrence of this tag.

3. Add Twitter Card Meta Tags

Right below this tag, create a new line and paste the following code:

	<!-- Twitter Card Tags -->
	<meta content='@yourTwitterHandle' name='twitter:site'/>
	<meta content='@yourTwitterHandle' name='twitter:creator'/>
	<b:if cond='data:post.firstImageUrl'>
	   <meta content='summary_large_image' name='twitter:card'/> <!-- summary_large_image or any other your card types -->
	   <meta expr:content='data:post.firstImageUrl' name='twitter:image'/> 
	<b:elseif cond='data:blog.postImageUrl'/>
	   <meta content='summary_large_image' name='twitter:card'/>
	   <meta expr:content='data:blog.postImageUrl' name='twitter:image'/> 
	<b:elseif cond='data:blog.postImageThumbnailUrl'/>
	   <meta content='summary' name='twitter:card'/>
	   <meta expr:content='data:blog.postImageThumbnailUrl' name='twitter:image'/> 
	<b:else/>
	   <meta content='summary_large_image' name='twitter:card'/>
	  <meta content='https://example.com/default-image.jpg' name='twitter:image'/>
	</b:if>

4. Customize the Meta Tags

Be sure to replace the following with your own details:
- @yourTwitterHandle in the twitter:site and twitter:creator meta tags.
- The twitter:image URL in the <meta content='https://example.com/default-image.jpg' name='twitter:image'/> line, which is the default image for posts without images.

5. Save and Test Your Template

Currently, the Twitter Card Validator may not be operational. Instead, you can test your blog’s Twitter Cards by sharing a link to your blog in a tweet and observing how it appears in the Twitter feed. Ensure that the link expands into the desired card format with the correct metadata and images.

6. Reapply When Changing Themes

Keep in mind that if you change your blog theme in the future, you’ll need to repeat this process because new themes overwrite the HTML template entirely.

Once your blog is set up for Twitter Cards, you can confidently share your Blogspot links on Twitter, knowing they will appear in a more engaging format.

Conclusion

By enabling Twitter Cards, you’ll make your Blogspot blog more appealing and clickable when shared on social media. Follow the steps outlined above to set up Twitter Cards, and remember to test using the Twitter Card Validator to ensure everything is working as expected. With these enhancements, your blog posts will have a more professional appearance and higher engagement potential when shared on Twitter.

The enduring value of Email: why it still matters even today

2020-12-14T00:00:00+00:00

Despite the rise of privacy advocates discouraging use of email, it remains a valuable communication tool with numerous benefits. Here’s why you should continue to embrace email in your digital communication strategy.

The nostalgic value of written communication

When did you last see someone post a letter in an inland or envelope in the old-fashioned way? Believe it or not but the art of “letter writing” has become an almost extinct skill these days. Email is the last remaining vestige of actual written communication, allowing for thoughtful exchanges while preserving records in an open format.

No true alternatives

A plethora of instant messaging (IM) apps have emerged over the years, including Skype, WhatsApp, Telegram and Discord. However, none can replace the functionality of email. It’s the only communication mode where the sender can respond with substantial content after taking a moment to think, rather than quickly “IMing” a few words through a mobile app.

Open standards

Email is not controlled by any private sector entity; it operates on open standards based on well-established protocols like IMAP, SMTP and POP3. In contrast, proprietary platforms like WhatsApp and Telegram rely on non-standard technologies that restrict interoperability.

Accountability through records

The ability to retain records makes people more responsible in their tone and language. Unlike “disappearing WhatsApp messages”, emails can be archived and preserved for decades or even centuries, allowing for accountability and clarity in communication.

Enhanced security options

Email can be secured with tools like VPN and plugins like Enigmail. Contrary to popular belief, email communication is not inherently insecure. Just as SSL/TLS enhances the security of HTTP, Enigmail can be used to digitally sign and encrypt outgoing emails, safeguarding sensitive information.

As you navigate your digital communication landscape, consider the unique advantages email offers. It may be time to re-evaluate how you communicate and leverage the benefits of email in your professional and personal life.

DIY Keyword research for small bloggers using absolutely free tools

2020-12-06T00:00:00+00:00

There are a plethora of keyword research tools available in the market such as SEMRush, ahrefs, KWFinder, etc. and while a subscription to them makes sense if you are a small company or startup, it definitely doesn’t for small bloggers who work with a limited budget.

But even if you have an absolutely zero budget, you can still perform decent keyword research and get the exact same results that these popular web services offer. However, you won’t be served that information on a platter and you will have to work for it and setup your own workflow after sourcing inputs or raw material from various freely available tools. You must also know a little bit of scripting/programming and be prepared to write some scripts in a language like python.

Each blogger or researcher may have a different workflow with respect to keyword research, mine involves the following two tools:

Both are excellent and robust freely available tools (though the latter gets you only a limited number of searches each day which is fine if you’re a small blogger who posts only few times a week).

In general, writing any article involves the following steps:

Determination of the topic and/or niche you want to publish about.
Research and gather information or sources about it (like books, wikipedia, online blogs, etc.)
Actually write the article.
Give it a decent or captivating headline.

Performing keyword research beforehand will help you with almost all these steps extensively. Firstly, it will help you determine which topic or niche you must blog about so that people actually find your content through search engines. In other words, you shouldn’t end up writing an excellent piece about a topic which:

Nobody is interested in (low search volume).
Has so high competition that it’s very difficult to rank on the first page.

These are the two areas which you must actively avoid if you want to blog for visibility or getting web traffic. Of course, there are hobby bloggers who post just for the sake of fun or writing passion and that’s fine too.

Coming to the process of niche determination using keyword research, it involves two tasks:

Find out which keywords related to our broader topic are currently in vogue or trending.
Find out which among those related keywords have a combination of good search volume and less ranking difficulty.

With Google Keyword Planner, both these tasks can be clubbed into one as it provides ways to both discover new keywords and getting search volume and forecasts for a given bunch of keywords (see below).

An alternative process of niche determination involves using ATP (answerthepublic.com) for the first task as it provides a richer set of related keywords which are questions or slightly longer phrases which have the potential for attracting long tail search volume. You then input these longer keywords provided by ATP to the second part of the Google Planner Tool (“Get search volume and forecasts”). But keep in mind that you can’t just lift the keywords from ATP and enter them directly to Google Keyword Planner tool!

Here is where scripting comes into picture. The CSV data exported by ATP has to be sanitized to the proper format (extra columns removed) so that it can be then entered to the Google Keyword Planner Tool. Scripting is also needed to store the output CSV from Google Planner to a proper database. Personally, I prefer sqlite but mysql is fine too. Google’s output CSV has almost everything you need, almost everything which the other subscription based services provide.

But the only catch is that it’s a DIY. You need to write your own program or script which imports this output to your database tables and stores data in the respective fields. “Avg. monthly searches” is quite apparent, you can use a long integer column called searches to store this. “Competition” tells you qualitatively whether it’s high/medium/low for that keyword, and “Competition (Index Value)” does the same quantitatively. It’s an index value ranging from 0 to 100 and often referred to as KD (Keyword Difficulty) in the other popular tools.

Once you’ve built your own workflow with automated scripts that involves importing and sanitizing data from ATP/Google, and storing it to a local database, you should be able to get what you want with a simple SQL query such as this:

select keyword from keywords where keyword like '%flask%' and searches >= 1000 and comp_idx < 30;

This will return all keywords in the python flask niche which has a good search volume (1000 or above) and less difficulty to rank (competition index is less than 30). You can then proceed to use those keywords for writing your article which will now have a greater visibility.

A paid tool like SEMRush or ahrefs would have given you the exact same results as this SQL query but your solution is custom developed and more importantly, suited to your own work-flow. The only difference is that your database won’t be having billions keywords at the start but it will grow organically and gradually as you keep writing articles and keep researching new content.

How to perform Microsoft OneDrive OAuth sign-in and authorization in a python web app

2020-11-14T00:00:00+00:00

The methods and examples given in this article are based on flask framework but they should apply to django or something similar too with a little tweak. Few weeks ago, I had landed myself on a project of similar nature and though I found several helpful articles and blog posts (such as this one), none of them explained this process in a simple but comprehensive manner, so I’m writing one myself.

The first and foremost step is to visit the Azure developer portal and register your app there. Just make sure that you select “All Microsoft account users” for supported account types and not just personal Microsoft accounts. This is needed in case you want your web app to perform OneDrive file transfers on behalf of the signed in user.

Make note of the client_id and client_secret values and store them somewhere as you’re going to need them while writing your app code.

Its also important to add the following permissions by clicking the “API permissions” link, and then add a scope for them by visiting the “Expose an API” link on the developer console screen:

API/Permissions	Type	Description
Files.ReadWrite.All	Delegated	Have full access to all files user can access
offline_access	Delegated	Maintain access to data you have given it access to
User.Read	Delegated	Sign in and read user profile

Also note that if you couldn’t set the supported account types to “All Microsoft account users” in the initial setup for some reason, you can later change it by updating the following XML setting after visiting the “Manifest” link on the app console:

"signInAudience": "AzureADandPersonalMicrosoftAccount",

Finally, its also important to configure redirect URIs by visiting the “Authentication” screen. This is the URL endpoint where our signed-in user will be redirected to and our app receives the “code” request argument which is needed to complete the OAuth process. In my case, I configured two redirect URIs, one for testing and one for production respectively:

- http://localhost:5000/post_onedrive
- https://example-app.com/post_onedrive

Once the registered app is configured thus, we focus on our python app. Its important to provide the two URL endpoints here:

First, where our signed in user visits (by clicking a link/button that says “Authorize OneDrive” or “Link OneDrive”), and thus starts the OAuth process:

@app.route('/link_onedrive', methods=['GET'])
def link_onedrive():
    conn, cursor = db.opendb()
    settings = cursor.execute("select * from settings").fetchone()
    url = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id={client_id}&scope={scope}&response_type=code&redirect_uri={redirect_uri}"
    #scopes = "wl.basic onedrive.readwrite wl.offline_access"
    scopes = "Files.ReadWrite.All offline_access"
    url = url.format(client_id=settings['od_client_id'], scope=scopes, redirect_uri=OD_REDIRECT_URI)
    print("url: ", url)
    return redirect(url)

In this case, I’ve already stored the app credentials like client_id, etc. in a sqlite table called settings from which I’m fetching them using a database connection. After that, its only a matter of calling the oauth URL with those specific parameters.

Second, where the OneDrive’s redirection is handled (/post_onedrive in above example):

@app.route('/post_onedrive', methods=['GET', 'POST'])
def post_onedrive():
    conn, cursor = db.opendb()
    settings = cursor.execute("select * from settings").fetchone()
    print("One Drive: HERE HERE!")
    print("now calling finish url")
    url = "https://login.microsoftonline.com/common/oauth2/v2.0/token?client_id={client_id}&redirect_uri={redirect_uri}&client_secret={client_secret}&code={code}&grant_type=authorization_code"
    data = {'client_id': settings['od_client_id'], 'client_secret': settings['od_client_secret'], 'redirect_uri': OD_REDIRECT_URI, 'code': request.args['code'], 'grant_type':'authorization_code'}
    url = url.format(client_id=data['client_id'], redirect_uri=data['redirect_uri'] , client_secret=data['client_secret'], code=data['code'])
    headers = {'Content-type': "application/x-www-form-urlencoded"}
    resp = requests.post(url, data=data, headers=headers)
    print("json response:", resp.text)
    obj = json.loads(resp.text)
    
    print("saving tokens")
    cursor.execute("delete from user_creds where user_id=? and auth_type='onedrive'", (session['user']['id'],))
    cursor.execute("insert into user_creds(user_id, auth_type, access_token, refresh_token) values (?,?,?,?)", (session['user']['id'], 'onedrive', obj['access_token'], obj['refresh_token']))
    cursor.execute("update users set od_auth='y' where id=?", (session['user']['id'],))
    session['user']['od_auth'] = 'y'
    conn.commit()
    flash("Your One Drive account was linked successfully!")
    return redirect(url_for("index"))

Once OneDrive redirects me to the /post_onedrive endpoint, I just have to complete the process using the code (request.args['code']) parameter, so that I can get access_token and refresh_token values. Once I get these values, I can store them as credentials against that user’s account (user_creds sqlite table in this case). Once that is done, I (or my app) can access that user’s drive any time. The access_token credential has a expiration limit of one hour but after that we can get a new access_token using the refresh_token parameter.

Here is a part of cron.py background script which keeps accessing each user’s drive by using the access_token and refresh_token received thus. In this case, the function is used to create a new folder on the user’s drive at a given path:

def create_folder_in_od(settings, creds, dst_path, folder_name): # create folder in onedrive
    write_log(0,0,"creating folder in onedrive: %s/%s" % (dst_path, folder_name))
    headers = {'Authorization':'bearer ' + creds['access_token'], "Content-Type": "application/json"}
    drive_url = 'https://graph.microsoft.com/v1.0/me/drive/root:%s:/children' % dst_path
    data = {'name': folder_name, "folder": { }}
    resp = requests.post(drive_url, data=json.dumps(data), headers=headers)
    write_log(0,0,"success!")

Here, settings and creds correspond to sqlite rows in settings and user_creds table respectively, which store the app settings (client_id and client_secret) and user settings (access_token and refresh_token) respectively.

The drive_url variable consists of two parts: The Microsoft Graph site URL which provides the Drive API (graph.microsoft.com/v1.0) and the particular API endpoint for creating a new folder on the drive (/me/drive/root::/children). Similarly, there are OneDrive API endpoints for doing other things, they are thoroughly documented, [here](https://docs.microsoft.com/en-us/onedrive/developer/rest-api/resources/driveitem?view=odsp-graph-online) you can find various API endpoints for uploading, downloading, etc.

In the next couple of days, I’ll try to put up a demo repo for this whole thing on Github. Until then, Happy Coding!

How to clear individual site cookies on android firefox

2020-09-05T00:00:00+00:00

One might wonder why an entire blog post is needed for a simple matter of clearing cookies on a web browser in the 21st century. But believe it or not, such are the state of things when it comes to android browsing! Firefox is a great browser to have on Android because it gives you three features which are impossible to get even with Google’s Chrome:

Plugin Support
Ad Blocking
Data Privacy

That’s why over 3 million users have decided to stick with firefox despite the fact that its performance totally lacks in comparison to Chrome. Even so, there are some annoying bugs in firefox such as not being able to clear cookies for individual websites (at least as of version 68.11). This seemingly simple feature isn’t available out of the box on firefox and when I googled for it, I found two solutions:

As this android stackexchange answer suggests, you can try extracting the below data file (where firefox stores cookies) to your PC, edit it and then put it back (you must be rooted though):
```
 /data/data/org.mozilla.firefox/files/mozilla/{YOUR_PROFILE}/cookies.sqlite
```
But if you aren’t rooted or you don’t want to go through the above troubled exercise, I’ve found a much simpler solution which is to install this extension called Cookie Manager by Rob W.

Since the android firefox browser supports extensions, I found it much easier to just install this extension, open it and type a domain such as *.reddit.com for which you want the cookies deleted, and press the “Search cookies” button:

Then you just click the “Select all” button at the bottom followed by “Remove selected”, that’s it!

SEO vs Content: What really drives traffic to your blog?

2020-08-16T00:00:00+00:00

The Content vs. SEO Debate

In the content publishing world, there has been a constant battle between the “Content is King” camp and the “SEO” camp! The first camp insists that it’s good content that essentially drives traffic to your blog or website. The second camp insists that it’s good Search Engine Optimization (SEO) techniques like keyword research and relevant meta tags that does it by standing up to the search engine algorithms that determine the page ranking for each link.

The truth, obviously, lies somewhere in the middle. There used to be a time in the 1990s (and probably early 2000s) when you could easily fool the Google search algorithm into making your page rank on top by merely spamming it with good keywords. But that started changing as the algorithms got smarter, today they are able to separate the wheat from the chaff. Today, they are able to tell the difference between good content and bad, or do they?

In a perfect world, SEO should work only to the extent of supply/demand of relevant information. In other words, to the extent that people searching for online booking of flight tickets is greater than people searching for flight simulator games, the content related to former must rank above for most queries related to “flights” and thus they should bag a larger number of “hits” overall than the latter.

How Search Engines Have Evolved

But among all the “online flight booking” websites, how does the Google algorithm decide which ones to put on top and which ones at the bottom? How does it know that makemytrip.com and goibibo.com should rank higher than cleartrip.com and air.irctc.co.in for an Indian visitor? Or for that matter, how does it know that expedia.co.uk and skyscanner.net should be placed higher than momondo.co.uk or wegotravel.co.uk for a UK visitor?

Now, this is where SEO and keyword research come into play which supposedly has some influence on that algorithm. SEO proponents like to think that their proposed measures (like filling the right meta tags, populating the page with right keywords, etc.) play a role in making makemytrip.com rank higher than cleartrip.com in Google searches. But can that really be true? Common sense suggests that search algorithms are a whole lot more complex beasts today than they once were and are influenced by just way too many things to create such a direct relationship.

The Role of Keywords and Links

Links (inbound and outbound). They play a large role in making the ranking decision. It could be the fact that makemytrip.com has a lot more links flowing in from highly reputable sites than air.irctc.co.in, so the algorithm is considering that in ranking. And of course, they should all be good (organic) links with no evidence of foul play! Now, trying to influence one’s own site’s ranking by throwing their links across various social media sites is an SEO technique called “link building” and is a slightly grey area. Do it too much and the search engines might ban your site or blog entirely in today’s age. Done correctly, link building provides a good initial boost of popularity, it’s a kind of marketing dude who goes across each home and tells good thinks about your product. But if the dude tries to be too intrusive or pushy, he might get blocked from the apartment. Thus, link building is more of a marketing trick than SEO technique in my opinion.
Keywords. Well, they sometimes do work but I’d like to think of them as the effect of a well ranked page, not the cause. In retrospect, it explains how the page’s content got ranked above its peers by the ranking algorithm but trying to replicate that strategy on another site by attempting to use more of those keywords may not always be successful. Trying to reduce textual content to mere keywords is like trying to reduce the human mind and emotions to mere neurons and synapses in the brain! They do play a huge role in the thinking process, but we humans are definitely more than our brain cells. That’s the problem with many of today’s so called “SEO Experts”, they are pushing keyword research as some kind of a silver bullet to page ranking while ignoring everything else.
Content quality. Ultimately, it’s good content that every googler is really searching for! It’s definitely not very easy to create good textual content and that’s because writing is not an easy craft. It takes a lot of effort to come up with good content periodically and this difficulty is what has given rise to the whole SEO industry! It’s my unpopular and controversial opinion that in an ideal world, SEO shouldn’t be needed at all (except to the extent of finding out what kind of content people are searching for (demand), so that you may write content relevant to that (supply)).

Why Content Quality Matters Most

Again, trying to meet supply with demand is a common sense discipline, I wouldn’t call it SEO at all! But you don’t really need keyword research to tell you that do you? If you are an industry veteran who has some experience under your belt, don’t you already know what is the trendiest subject in your industry? As a php programmer, for example, you must know that Laravel and Symfony are far more trendy these days than CodeIgniter or CakePHP. That doesn’t mean that you shouldn’t write articles on CodeIgniter, by all means you should if that’s your area of expertise! But you must also be aware that those articles won’t be catching as many eyeballs as the ones on Laravel or Symfony.

Same is the case in regard to Angular, React and Vue.js in the JavaScript world. You can probably rank much higher by writing on any of these frameworks instead of something like Backbone.js or Ember.js. All it takes is a little bit of reading and presence on social media to know that the former topics are in lot more demand than latter. Consequently, if you are a blogger who writes for adsense income or a freelance copywriter who sells content for a living, you’ll know better to write more on the former topics than the latter. On the other hand, if you are a hobbyist blogger who writes for self-fulfillment and your interest in Backbone/Ember, you will write about the latter subjects even if they are less in demand.

Conclusion

The debate between the “Content is King” and “SEO” camps will probably continue for decades to come, but it all depends on who you are and what you hope to gain from SEO (or some of it’s aspects called content marketing). If you are a blogger or writer, my advice for you is to focus on mastering the art of churning great content frequently than trying to master the art of SEO.

How to use window.hashchange event to implement routing in vanilla javascript

2020-08-06T00:00:00+00:00

The popular wisdom when it comes to implement routing in your client side apps or single page apps (SPA) as they are called these days, is to just grab an “off-the-shelf” framework like angular or vue and start using it. Well, that’s the easy way out but you will never understand the nuts and bolts of how things like routing work at the low level. Besides, if routing is the only major requirement in your app, why riddle your app with unnecessary bloat of frameworks?

If you happen to agree with that wisdom, you can proceed further to learn how exactly to implement routing in vanilla javascript.

When it comes to routing, your app has two main options:

To use hash urls such as http://localhost/#login and http://localhost/#register
To use fully fledged or “proper” urls like http://localhost/login and http://localhost/register

As you can see, the major difference is that the former method uses hashes (#) which are used to control routing on the client side whereas the latter method uses complete urls without the hashes. The latter method requires you to use the history.pushState API method and while these urls look quite sexy and elegant, they come with a cost: your backend web server must support routing up to multiple path fragments too. This means that you will need a “proper” web-server on the backend such as apache or nginx, you can’t do this with “simple” http servers such as the python’s built-in http.server module or even static hosting facilities like github-pages and netlify. So, if you intend to host your app using github-pages or netlify, you must remember to use hash urls only!

Considering that hash urls method is compatible with all kinds of backend servers, its a good practice to use them in all apps, irrespective of the backend used. Once again, popular wisdom may not agree with that and most “experts” may in fact advise you to do the opposite!

Coming to the point, implementing hash urls is as easy as handling the window.hashchange() browser event. Using vanilla javascript, it can be as simple as this:

window.addEventListener('hashchange', function(){
	switch(location.hash) {
		case "#index":
			doIndexStuff();
		case "#login":
			doLoginStuff();
		case "#register":
			doRegisterStuff();
		default:
			handleDefaultCase();
	}
});

You see how simple url routing is with hashchange? You want to have advanced routing features like path fragments and parameters? No problemo! You can use simple string.split() method to take care of that:

var url = location.hash.substring(1); //#user/prahlad?view=profile
var parts = url.split("?");
var fragments = parts[0].split("/"); // user/prahlad
var paramParts = parts[1].split("&"),
	params = [];
for(var i=0;i<paramParts.length;i++) {
	var pair = paramParts[i].split('='),
		obj = {};
	obj[pair[0]] = pair[1];
	params.push(obj);
}

This simple piece of code will parse your hash url and fill the successive path fragments (user/prahlad) in the fragments array and the parameter pairs (?view=profile) as key-value objects in the params array. You can of course mix, match and customize this code as per your own scenario but this is all there is to routing in about 90% of use cases!

Enjoy and Happy Coding!

Two years later, the reasons for Microsoft's Github acquisition still remain a mystery

2020-07-28T00:00:00+00:00

More than two years have passed since I shared my perspective on Microsoft’s Github acquisition in this blog post earlier, and the point still stands today. Why did Microsoft decide to acquire a startup company much smaller in size which had no viable and long term revenue/profit generation model?

If Microsoft wanted their team to use Github in their daily work-flow, there was hardly a need to acquire the whole company in a multi billion dollar deal. Github offered free private repositories even back then (and even their premium-tier subscription should have cost peanuts for Microsoft to get it for their entire staff!).

If Microsoft wanted to sell it to enterprise clients, again much better and creative ways to do that. Brand Microsoft is several times more valuable than Brand Github in the enterprise world anyway. And those are the type of clients who would have already invested heavily in Microsoft stack, so something like Microsoft’s Team Office would make a lot more sense for them than Github.

What then could have been the reason? Its still unknown but that’s probably true about Microsoft’s many other past acquisitions including Skype, Hotmail, Nokia and Linkedin. What exactly did Microsoft gain by each of these acquisitions? Arguably none in financial terms but perhaps that wasn’t their motive at all. Perhaps they want to make a mark on the sands of time by owning some of the most popular artifacts of our times? That could also explain Microsoft getting themselves a costly seat on the board of Linux Foundation too.

Update

Some interesting perspectives from reddit about this:

A few years ago I was at a conference ’Dev ops days’ where Microsoft had a presence. The Microsoft speakers were part of Microsoft’s Unix/Linux team’s. MS speakers talked about MS investing millions into Open source as they see it as the future. Further they discussed their teams Git commits on open source projects.

STAMINA comes to mind. A lot of “hack” code ends up on GitHub and occasionally the bits and pieces that can be turned into malware. I wouldn’t be the least bit surprised if this is so they can use code as it appears to run through this and check code on the fly.

Listen to this : https://twitter.com/i/broadcasts/1OyKAYWPRrWKb Especially the bit where their AI is able to code. Any guesses how it learnt that?

Seems a bit of a stretch to buy GitHub to train a coding AI, MS do have access internally to several decades worth of code, and of course it’s pretty simple to scrape code for free from all manner of open source projects and Github.

A big part of the code of the projects at Github is freely accessible. Therefore, in my opinion, Microsoft would not have had to buy Github to train the AI.

They are developing VS Code and MS Terminal on github with the github community. They wanted to secure the platform so no one could “take” it, or make it difficult for them. (Edit: it = the platform and access to the community)

Intro to message flashing: A handy way to send messages across page requests in Flask

2020-07-08T00:00:00+00:00

Message flashing is a very handy technique you must be aware of if you are a Flask developer. A common recurring pattern in web development is to send messages across web requests, especially in case of redirects when the route or controller that redirected wants to display some basic text message (such as “Login Successful!”) on the target page after redirection.

Let’s look at this simple example app which asks for a username/password, then redirects user to a special page (“/homepage”) if the validation was successful.

from flask import Flask, request, redirect, url_for

app = Flask(__name__)

@app.route("/homepage")
def homepage():
    message = "You have been logged in!"
    return message + "<br>Welcome to home page"
    

@app.route("/", methods=['GET','POST'])
def index():
    if request.method == 'POST':
        if request.form['username'] == 'admin' and request.form['password'] == 'admin':
            return redirect(url_for("homepage"))
        else:
            return "Incorrect username/password."
    elif request.method == 'GET':
        return """
<!doctype html>
<html>
<head></head>
<body>
    <form method="POST">
        <input name='username' placeholder='Username' required> <br>
        <input type='password' name='password' required> <br>
        <button type="submit">Login</button> <br>
    </form>
</body>
</html>
"""
	
if __name__ == "__main__":
	app.run(debug=True)

Now, there are two basic problems here. Firstly, the home page always displays “You have been logged in!” but what if I want to display it conditionally (i.e when the user has just performed a successful login) and not each time? Secondly, there is also the case that I may want to customize this message (with a user greeting, for example).

Before knowing about the message flashing technique, I used to use the cumbersome technique of storing such custom values to the session object, then reading the session object in the corresponding target route (“/homepage”). But now, I’ve found a much simpler and easier way which is message flashing. All you have to do is add an import to flask.flash and flask.get_flashed_messages class methods in your code and call it just before a redirection:

from flask import flash
...
...
...
flash("You have been logged in!")
return redirect(url_for("homepage"))

You can also add a second parameter to flash call which is the category it is meant for (such as error, info, etc.). Once you flash a message, all you must do on the target route (“/homepage”) is fetch the last flashed message like this:

message = get_flashed_messages()[0]
return message + "<br>Welcome to home page"

Of course, if you happen to use a jinja2 template for displaying output then get_flashed_messages() is directly callable in that too:

{% with messages = get_flashed_messages(with_categories=true) %}
  {% if messages %}
    <ul class=flashes>
    {% for category, message in messages %}
      <li class="{{ category }}">{{ message }}</li>
    {% endfor %}
    </ul>
  {% endif %}
{% endwith %}

After playing with message flashing for some time, I came to know that it also uses the session object internally. For example, it won’t work if you omitted the app.secret_key attribute before starting the app. As I said earlier, all of the above can be done using sessions too but message flashing is a more efficient way of doing it and requires less code. And the end, finding efficent ways of achieving things is what we programmers should be striving for, enjoy!

How to use robocopy for taking regular backups on windows

2020-07-08T00:00:00+00:00

Having switched from linux to windows recently, I was looking for an rsync alternative. rsync is a popular tool in the linux world used for taking routine backups and also general file transfer. But since I use it mostly for backups and I’m mostly a command line dude, I was trying to find a windows equivalent which can do the same. That’s when I came across the robocopy command.

Despite its trivially sounding name, robocopy.exe is a versatile tool found in almost all windows installations at C:\windows\system32 and does a great job of taking incremental file backups. The best way to configure a robocopy backup is to create a simple batch file called backup.bat such as the one below:

@echo off
if not exist "e:\source\" (
	echo "backup drive not found. make sure e:\ corresponds to backup usb drive"
	goto :halt
)
echo Welcome!
echo ***

robocopy c:\source e:\source /E /PURGE /ndl /Z
:halt

This example assumes that you have a directory located at c:\source and you want to backup that entire directory to another drive (for example, a USB pen-drive) at e:\source. If you have more directories to backup, you can add them before the :halt statement like this:

robocopy c:\docs e:\docs /E /PURGE /ndl /Z
robocopy c:\music e:\music /E /PURGE /ndl /Z

Now, let’s look at what those parameters to robocopy command do. The source and destination folders are obvious, here is what follows:

/E		Copy subdirectories (similar to --recursive option in rsync)
/PURGE	Delete destination files that no longer exist in source (--delete rsync option)
/ndl	Don't log directory names while copying
/Z		Copy files in restartable mode

You can run robocopy /? to see a full list of options available in this tool. If you want to schedule backups (like once a week or month), simply go to Task Scheduler from Control Panel -> Administrative Tools and schedule the above script to run accordingly.

Enjoy, your DIY backup solution for Windows is ready!

A crash course in python packaging

2020-07-02T00:00:00+00:00

This guide isn’t for the newbie who is just learning python programming (they are better off doing a “proper” reading of the official docs instead). This is for the seasoned coder who dabbles in multiple technologies and needs a quick refresher on how to go about building a neat zip or tar.gz package from their existing python code, and optionally upload it to python package index (pypi).

First of all, its necessary to encapsulate your individual python scripts into a python package. Individual *.py scripts can’t be packaged on their own, you must add them to a directory which will represent your package. For example, the foobar package directory is as follows:

...foobar\
	...main.py
	...util.py
	...__init__.py
	...etc.

The __init__.py is needed so that python treats your directory as a package. It can be either left totally blank or consist of helpful imports if you need them. The rest of your code also needs to be adjusted as every module is contained in a package now. So for example, the following main module won’t work because the imported module (util) resides in a package now.

import util

if __name__ == "__main__":
	print(util.print_line())

This code must be re-factored so that it imports the util module from the foobar package instead:

from foobar import util

Of course, this must be done in each module. When your package gets installed on the user’s machine, the scripts will be inside a directory such as following:

C:\Users\<username>\AppData\Local\Programs\Python\Python36-32\Lib\site-packages\foobar\

And the user may be running the scripts from any random directory such as C:\source and there won’t be a util.py on their current path obviously! Hence, the module must be referred to by its global package naming convention. Since foobar package will be installed on the user’s machine, the above directory will be recognized by the python interpreter as foobar package directory where it will find util and other installed modules.

Once you do that, you must create a setup.py in the root directory of your source, so your directory structure becomes like this:

setup.py
...foobar\
	...main.py
	...util.py
	...__init__.py
	...etc.

If you manage multiple projects in your source directory, you can put this whole bunch of files in a “foobar_project” or similarly named directory for organization.

You may add many other files at the root level such as MAINFEST.in which is only needed in case you want to add configuration or other files from different locations to your setup file. There could be other files too such as README.md for your project’s description and your project’s LICENSE file.

The contents of the setup.py should be as follows:

#!/usr/bin/env python3
import foobar
from setuptools import setup, find_packages

s = setup(
	name=foobar,
	version="0.1",
	license="MIT",
	description="Dummy package called foobar",
	long_description="Dummy package called foobar, some longer description.",
	long_description_content_type='text/markdown',
	url='https://github.com/prahladyeri/%s' % pkg_name,
	packages=find_packages(),
	include_package_data=True,
	entry_points={
		"console_scripts": [
			"foobar = foobar.main:main",
		],
	},
	install_requires=['requests', 'colorama'],
	python_requires = ">= 3.4",
	author="Prahlad Yeri",
	author_email="prahladyeri@yahoo.com",
	classifiers=[
		"Programming Language :: Python :: 3",
		"License :: OSI Approved :: MIT License",
		"Operating System :: OS Independent",
	],
	)

The setup function takes some arguments which is basically the information needed to build and install your package. The most important argument is name, this is how your package will be identified by the pip installer and the python packaging index (pypi). For things like version number, license, description, etc., its a better idea to define those variables in __init__.py itself, so that you can directly import them like this instead of hard coding:

from foobar import __version__, __description__, __license__

An important parameter here is entry_points. You must define the command line script name here which your user will use to run your script. If its going to be foobar, you must also specify which module’s function must be called as entry point. In this case, when user runs foobar, the package foobar’s main module will be called and from that, the main() function will run:

	entry_points={
		"console_scripts": [
			"foobar = foobar.main:main",
		],
	},

Of course, you must refactor your code so that it cannot simply start running from “main” check but must be placed in a separate main function:

def main():
	pass # <your code goes here
	
if __name__ == "__main__":
	main()

You can also pass command arguments arguments through this arrangement but its better to use the argparse module for that instead of parsing the CLI arguments manually. Once you’ve done this wiring of your code properly as described, you then have to just run the following command to test your build by going to the package root directory (where setup.py is located):

python setup.py sdist

The sdist argument creates a source distribution (typically tar.gz file) under the dist subdirectory. It will be of the format:

foobar-<version>.tar.gz

Here, is whatever you provide in the setup argument. You can test the installation of this build by running:

pip install dist\foobar-<version>.tar.gz

This must install your package on the local machine, it will have the same effect as if your package was installed by pip itself:

pip install foobar

Of course, you cannot run the above command until you upload your foobar package to the python package index (pypi). If you don’t want to do that and just distribute zipped installation files to your users/clients, what you’ve learned until now should suffice. But if you want to upload your own package to pypi, then you must install a dependency library called twine for that:

pip install twine

Now you are ready to upload your own package to python package index (pypi). For that, you must first register an account at python package index (pypi). After that, its as easy as running the twine upload command!

twine upload dist\foobar-<version>.tar.gz

The above command asks your pypi login credentials you just registered and then upload your package. Within a few minutes, your package will be available for the whole world to install through pip!

pip install foobar

Seamlessly migrate disqus comments to WordPress: a step-by-step guide

2020-03-11T00:00:00+00:00

If you’re switching from a platform using Disqus to WordPress, migrating your comments can seem daunting. Disqus was once a popular choice for blog comments, but with WordPress’s built-in commenting system and more robust plugins, many are making the switch. In this guide, I’ll walk you through how to easily migrate your Disqus comments to WordPress using updated scripts and tips to ensure a smooth transition.

This is an updated follow-up to my previous guide on Importing Disqus Comments into WordPress. The process remains largely the same, but I’ve made tweaks and improvements to make it easier for first-time users. Let’s dive in!

Why Migrate from Disqus to WordPress?

Disqus was once a go-to for comment management, but its reliance on third-party services and recent issues like increased ads and slow performance have prompted many bloggers to look for alternatives. Migrating to WordPress’s native system means faster load times, more control over your data, and fewer dependencies on external services.

If you’re ready to switch, follow this guide to migrate your comments efficiently.

Step-by-Step: Migrating Disqus Comments to WordPress

I’ve updated the migration scripts to reflect changes in both WordPress and Disqus. Below is a breakdown of the process, as well as the script updates that make it easier.

1. Adjusting the URL Matching Logic

The previous migration process required exact URL matches between your Disqus and WordPress posts, which could cause issues. I’ve now optimized the script to compare only the URL paths (ignoring domains) using the parse_url function:

$cpath = parse_url($comment['url'], PHP_URL_PATH);
$tpath = parse_url($t_url, PHP_URL_PATH);
if ($cpath === $tpath) {
    ...
}

This change makes it easier to match URLs across different domains, streamlining the migration process.

2. Handling Comment Flooding Errors

WordPress sometimes throws a “You are posting comments too quickly” error when bulk importing comments. To mitigate this, I’ve added a $sleep_interval parameter (default value is 2 seconds) to slow down the import:

$sleep_interval = 2;

If you encounter this error, you can increase the $sleep_interval or temporarily disable the comment flood protection by adding the following line to your functions.php file (remember to revert it after the migration):

add_filter('comment_flood_filter', '__return_false');

3. Parsing Comment Author Names

Sometimes the Disqus comment author’s name is stored in an array. The updated script accounts for this with a simple check:

if (is_array($comment['name'])) {
    $comment_author = (string)$comment['name'][0];
} else {
    $comment_author = (string)$comment['name'];
}

This prevents issues when migrating comments from authors with complex names.

4. Passing the Avoid Die Parameter to `wp_new_comment`

In some cases, the wp_new_comment() function would stop execution if there were issues with a comment. By passing the $avoid_die parameter as true, we ensure the script continues to run smoothly:

wp_new_comment($comment_data, true);

This change helps prevent the script from stopping prematurely.

Updated Migration Scripts

The latest versions of the scripts can be found below. These updates ensure smoother migrations with fewer errors and better compatibility with different WordPress setups.

console.php
disqus_parse.php

Place these files inside a plugin directory such as /wp-content/plugins/your-plugin/ and run them from there to begin the migration process.

Final Thoughts

Migrating comments from Disqus to WordPress might seem complex, but with these updated scripts and adjustments, you’ll have your comments fully migrated in no time. WordPress offers better flexibility, performance, and security when compared to third-party comment systems, and this switch is a great step toward streamlining your blog management.

Why you should never use the dict.get() method in python

2019-11-28T00:00:00+00:00

There is an increasing trend towards using the dict.get() method in Python, it’s very widely used in many tutorials and examples too. Consider the below example for instance:

>>> dd = {'a': 1}
>>> dd.get('a')
1

This simple dictionary dd has only one key (‘a’) and its corresponding value is 1. The proper way of accessing this key is to refer it as dd['a'] but you need to be sure that the dictionary has that key otherwise it’ll throw an error! Hence, as a shortcut way, programmers have started using dd.get() method instead. This method simply returns the key value if it’s present and None if there is no such key in the dictionary.

Seems good enough, right? Well, don’t fall into that trap so soon! In the real world, you’ll come across a situation sooner or later when you’ll have to handle a dictionary key who’s value is None! What will you do then? Consider this example:

>>> dd = {'a': None}
>>> dd.get('a')

In this case, you’ll never know whether a key was present in the dictionary or one was there and its value was None. Knowing the difference could be crucial in some situations and not doing so might cause subtle bugs in your program which will be very hard to trace later. The best habit to access a dictionary is this:

>>> if 'a' in dd:
>>>     x = dd['a'] # do whatever
>>> else:
>>>     pass # handle this

Or you can even simply refer its key directly if you are sure that the key exists:

>>> x = dd['a'] # do whatever

Similarly, there are two different ways to remove an item from dictionary but there is nothing wrong in using either of them (except of course that you need to be sure that the key is present and use an exception handler if you aren’t):

>>> dd.pop('a')
>>> del dd['a']

A complete guide to identifying your MTP device on Linux using bash

2019-10-27T00:00:00+00:00

In the good old days, USB file-transfer between your laptop and smart-phone used to happen through the much easier “mass storage” mode which works just as if you had inserted an external pen-drive. But these days, most smart-phones and tablets have shifted to the MTP (Media Transfer Protocol).

Unlike a mass storage device, the MTP device doesn’t show up in the usual /media/<user>/<device-label> folder in a linux distro which makes backup automation or other file transfer tasks more difficult as you cannot simply hard-code this path and write bash scripts based on that.

This isn’t possible in case of MTP because the label is dynamically determined using two important bits of information called BusID and DeviceID, and the file-system path works out to be something like this:

/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C010%5D/Internal shared storage

In this example, 001 is the BusID whereas 010 is the DeviceID. You can find these two values by running the lsusb command before and after the MTP device is inserted:

However, these two numbers (BusID and DeviceID) are dynamically determined depending on when and where you inserted the MTP device. Hence, if you are writing an automation script for backup, etc., you need to calculate it using other two static bits of information which are always constant for your device, they are called VendorID and ProductID.

In this example (for my Redmi Y2 phone), they happen to be 2717:ff40 (displayed right next to the BusID and DeviceID in above screen). Now that you have these two static bits of information which are always going to be in a given format (VendorID:ProductID), you can pass it to a bash function to determine the path dynamically:

function get_mtp_path() {
	str=$(lsusb|grep "$1")
	bus=`echo $str | awk '{split($0,a," "); print a[2]}'`
	dvc=`echo $str | awk '{split($0,a," "); print a[4]}' | sed 's/://g' `
	echo "mtp:host=%5Busb%3A$bus%2C$dvc%5D"
}

redmi_y2="2717:ff40"
redmi_y2_path=`get_mtp_path "$redmi_y2"`
folder="/run/user/1000/gvfs/$redmi_y2_path"

In above script, $redmi_y2 variable is hard-coded which is then passed to the get_mtp_path bash function to determine the actual path which is then stored in the $folder variable.

Now that you have $folder, you can do anything with it. For example, you can use rsync to backup your smart-phone pics in the DCIM folder to your laptop:

rsync -vrt "$folder/Internal storage/DCIM/" "~/Pictures/"

Alternatively, you can run a normal backup of your Documents folder to your smart-phone:

rsync -vrt  "~/Documents/" "$folder/Internal storage/Documents/"

The possibilities of this are immense!

Python recipe: Combine multiple images into one PDF

2019-10-24T00:00:00+00:00

Many a times, you come across the need to combine multiple image files (.jpg, .png, etc.) into one single portable document format (.pdf). Maybe, you have a bunch of handwritten notes which you want to organize into one file. Doing that is very trivial if you know python. To being with, you must have the fpdf library installed:

pip install fpdf

The first thing you do is create the FPDF() object and set the source directory where you have the images stored. sdir is that variable. You also need to set two variables to capture the height and width of the images respectively.

import os
from PIL import Image
from fpdf import FPDF
pdf = FPDF()
sdir = "imageFolder/"
w,h = 0,0

Then comes the looping part. This makes things easier if you have the image files named serially in numeric order (IMG001.png, IMG002.png, etc.). Otherwise, you can adapt this code to manually or sequentially process them one after another by setting the fname variable:

for i in range(1, 100):
	fname = sdir + "IMG%.3d.png" % i
	if os.path.exists(fname):
		if i == 1:
			cover = Image.open(fname)
			w,h = cover.size
			pdf = FPDF(unit = "pt", format = [w,h])
		image = fname
		pdf.add_page()
		pdf.image(image,0,0,w,h)
	else:
		print("File not found:", fname)
	print("processed %d" % i)
pdf.output("output.pdf", "F")
print("done")

The output of the above process will be stored in a single “output.pdf” file. So, that’s pretty nice and easy, isn’t it? Here is the entire code.

import os
from PIL import Image
from fpdf import FPDF
pdf = FPDF()
sdir = "imageFolder/"
w,h = 0,0

for i in range(1, 100):
	fname = sdir + "IMG%.3d.png" % i
	if os.path.exists(fname):
		if i == 1:
			cover = Image.open(fname)
			w,h = cover.size
			pdf = FPDF(unit = "pt", format = [w,h])
		image = fname
		pdf.add_page()
		pdf.image(image,0,0,w,h)
	else:
		print("File not found:", fname)
	print("processed %d" % i)
pdf.output("output.pdf", "F")
print("done")

How to implement URL routing in vanilla JavaScript

2019-07-18T00:00:00+00:00

In this post, I’ll show you how to implement dynamic URL routing in vanilla JavaScript without using any heavy frameworks like angular, vue, react, ember, etc. in less than 30 lines of code!

The important attributes you need to know are window.location and more specifically to our purpose, window.location.hash. This built-in property basically tells us what page URL or route we are presently in (ex: index.html#something). This property is blank if there is no hash URL and you are on the main URL (ex: index.html or index.html#). In that case, you may assume a default hash such as _index to maintain consistency:

function displayHash() {
  var theHash = window.location.hash;
  if (theHash.length == 0) { theHash = "_index"; }
  var elems = document.querySelectorAll("#caption");
  elems[0].innerText = "Current Hash: " + theHash;
  return true;
}

This simple function displays the current hash in a heading element named #caption. If you want this to fire each time the hash changes (user navigated to a different page in your app), you can do that using the hashchange window event:

window.addEventListener("hashchange", function() {
  console.log("hashchange event");
  displayHash();
});

And finally, in order to display the heading initially when the user first loads the URL in their browser, you can call displayHash() in the DOMContentLoaded event call too:

window.addEventListener("DOMContentLoaded", function(ev) {
  console.log("DOMContentLoaded event");
  displayHash();
});

That’s all folks! URL Routing is so easy to implement without using any heavy and bloated frameworks at all. The demo for this example can be found at prahladyeri.github.io/learn-js/url-routing. You can see that as you click the individual hyper-links, the heading label changes to display the current hash.

The complete source code for this can be found at my github repo, prahladyeri/learnjs (its less than 30 lines ;-).

I came across this solution through this StackOverflow post by Tulio Faria which uses jquery but I adapted it for vanilla JavaScript.

The Agile Manifesto's shortcomings: addressing modern development needs

2019-07-02T00:00:00+00:00

The Agile Manifesto has been a cornerstone of software development since its creation in 2001, promoting flexibility, collaboration, and the prioritization of working software over comprehensive documentation. However, as development environments and business needs have evolved, many professionals are beginning to recognize certain limitations in the Agile approach, particularly in how it addresses modern complexities and the scale of today’s projects.

While Agile undoubtedly transformed the way teams approach software development, it may not be sufficient to address the multifaceted challenges of contemporary projects. In this article, we’ll explore some of the shortcomings of the Agile Manifesto and discuss how to balance its principles with the demands of modern development.

1. Overemphasis on “Working Software” Over Documentation

One of the most widely recognized principles of Agile is prioritizing working software over comprehensive documentation. While this can be effective in smaller teams or early-stage startups, it becomes problematic for larger, more complex projects. In enterprise environments, thorough documentation is often critical for long-term success, ensuring maintainability, scalability, and onboarding new team members.

Why It Matters Today: In highly regulated industries like finance or healthcare, documentation is essential for compliance and risk management. Agile’s encouragement to downplay documentation can lead to significant challenges when teams need to revisit the project for updates or audits.
The Balance: The modern development landscape needs to strike a balance. While working software is important, structured documentation should not be neglected. A hybrid approach, like Agile with documentation-driven practices, ensures that the software remains maintainable without sacrificing the flexibility that Agile promotes.

2. Undervaluing Processes and Tools

Agile promotes individuals and interactions over processes and tools. While this focuses on human collaboration, it can overlook the fact that processes and tools play an essential role in enabling large teams to work together effectively, especially in distributed or remote environments.

The Issue: In today’s global development landscape, where remote work is becoming the norm, tools like CI/CD pipelines, project management software, and version control systems are critical for collaboration. Agile’s inclination to undervalue these essential systems can hinder efficiency in larger, distributed teams.
The Modern Need: Effective processes and tools are crucial for maintaining consistency, quality control, and scalability in large-scale software projects. Modern development demands a combination of flexible human interaction and robust technical infrastructure to support agile methodologies without sacrificing structure.

3. Client Collaboration vs. Contract Negotiation

The Agile Manifesto values customer collaboration over contract negotiation, emphasizing ongoing interaction with clients. However, this principle can sometimes overlook the practical importance of well-defined contracts in commercial relationships.

Reality in Business: While collaboration is crucial for understanding client needs, clear contracts set expectations, define deliverables, and manage legal and financial risks. Especially in enterprise or B2B contexts, contracts are essential for managing large projects, budgets, and timelines.
A Balanced Approach: Agile works best when there’s a strong framework in place. Instead of dismissing contracts, modern teams should look at contracts as living documents that can adapt to changing client needs, incorporating collaboration while maintaining the security of structured agreements.

4. The Danger of Over-Flexibility: Responding to Change Without a Plan

Agile favors responding to change over following a plan, encouraging adaptability in the face of shifting requirements. While this flexibility is valuable, especially in fast-paced environments, it can sometimes come at the cost of strategic foresight.

Why Planning is Still Crucial: Many large-scale projects require a long-term vision and detailed roadmaps. Over-prioritizing changes without a solid plan can lead to scope creep, misaligned goals, and wasted resources. Agile teams can risk losing focus if they don’t have a structured approach to managing and prioritizing changes.
Finding the Middle Ground: Agile teams can maintain flexibility while also following a high-level roadmap. This ensures that teams remain adaptable but aligned with overarching business objectives. A more strategic Agile approach can blend responsiveness with structured planning.

5. The Limits of Agile in Large-Scale and Regulated Industries

Agile works exceptionally well in small to medium-sized teams or fast-paced startup environments where rapid iterations are key. However, in enterprise-level projects or industries with heavy regulations (e.g., automotive, aerospace, or government projects), Agile may fall short.

Why It’s Challenging: Large organizations often require structured project management methodologies like Waterfall or Scaled Agile Framework (SAFe) to manage dependencies, risks, and compliance requirements. Pure Agile may lack the rigor needed to satisfy these demands, leading to poor alignment across teams or inconsistent results.
Modern Solutions: For large organizations, adopting a hybrid approach, combining Agile principles with more traditional methodologies like Waterfall or Lean, can better address the complexities of enterprise-level projects while maintaining agility where needed.

Conclusion: Adapting Agile to Meet Modern Needs

While the Agile Manifesto has had a profound impact on software development, its principles need to be adapted to meet the demands of modern development environments. Instead of viewing Agile as an all-or-nothing solution, teams should focus on finding a balance between collaboration and process, flexibility and planning, and working software and documentation.

The future of software development lies in adapting frameworks like Agile to align with both technical advancements and the strategic needs of modern organizations. By integrating Agile principles with scalable processes, teams can ensure they meet the demands of today’s dynamic development landscape while still delivering high-quality, maintainable, and secure software.

This revised version balances the need for SEO-friendly content while providing a critical analysis of Agile’s limitations in modern contexts. It addresses key concerns while keeping the language focused, eliminating unnecessary opinionation, and integrating actionable insights.

Introducing gh_announce: Automate github release announcements on twitter with this python bot

2019-06-27T00:00:00+00:00

I happen to maintain a lot of python projects on github such as distroverify and vtscan. And each time I make a tagged release on Github, I have to make a status tweet like this to let people know:

So today I thought why not automate this process by writing a python bot that:

Runs as a background cron on my computer.
Keeps checking my github release tags.
Post a new tweet in the above format for me whenever there is a new release.

The result of this endeavor is gh_announce. I used the popular tweepy library for the twitter api and my own written package cfgsaver to handle the configuration data such as my github username, twitter api credentials, etc.

Once that is done, its only a matter of cruising through the github user events api and check whether a new tagged release has come:

def check_activity():
	url = "https://api.github.com/users/%s/events" % config['github_username']
	resp = requests.get(url)
	acts = json.loads(resp.text)
	if len(acts) == 0:
		print("Zero events found, is this the correct github repo I'm looking at?")
		print("Run the program again with --config parameter to set the correct values")
		return
	for i in range(len(acts)):
		act = acts[i]
		if act['type'] == 'CreateEvent': #latest tag
			payload = act['payload']
			if payload['ref_type'] != 'tag':
				continue
			repo = act['repo']
			repo_url = "https://github.com/" + repo['name']
			tag_name = payload['ref']
			
			dt = parse_date(act['created_at'])
			delta = datetime.now() - dt
			days = delta.days
			hrs = delta.seconds // 3600
			mins = (delta.seconds // 60) % 60
			
			if delta.days >= 2: #this push is more than two days old, so just ignore
				continue
				
			tweet = False
			#check local config data to know whether we've already tweeted for this release
			if not 'pushes' in config:
				pushes = [act['id']]
				tweet = True
			else:
				pushes = config['pushes']
				if act['id'] in pushes:
					pass #do nothing
				else:
					tweet = True
					pushes.append(act['id'])
			if tweet:
				try:
					tw_announce(tag_name, repo['name'], repo_url)
				except Exception as ex:
					print("Error occurred: ", str(ex))
			config['pushes'] = pushes
			cfgsaver.save(pkg_name, config)

As you can see, I’ve ignored the pushes which are more than two days old as they may already have been tweeted or discussed, and can be ignored on the first run. For making the actual tweet, I use the tweepy library’s OAuthHandler for authentication, then call the api.update_status() method to make the actual tweet.

def tw_announce(tag_name, repo_name, repo_url):
	ss = "I have" if (config['Full_Name'] == None or config['Full_Name'] == "") else config["Full_Name"] + " has"
	ss += " just released version %s of %s on Github. Check it out! %s" % (tag_name, repo_name.split("/")[1], repo_url)
	#print("TWEETING: " + ss)

	if config['twitter_consumer_api_key'] == "":
		print("twitter api credentials missing")
		return
	auth = tweepy.OAuthHandler(config['twitter_consumer_api_key'], 
		config['twitter_consumer_secret'],
		)
	auth.set_access_token(config['twitter_access_token'], config['twitter_access_token_secret'])
	api = tweepy.API(auth)
	api.update_status(ss)
	print("successfully updated status for repo: %s, tag: %s" % (repo_name, tag_name))

You need to register your own app on https://developer.twitter.com/apps and get these four credentials if you intend to use the tweepy library for posting tweets on your behalf:

'twitter_consumer_api_key',
'twitter_consumer_secret',
'twitter_access_token',
'twitter_access_token_secret',

The app prompts for doing this on the first run of course. The wording of the tweet is kept in a manner that takes care of whether the Full_Name configuration setting is entered. If its there, the status starts with “Full_Name has just released version tag on Github…”, otherwise as “I have just released version on Github...". So, you can leave `Full_Name` setting blank depending on how you want the status structure to be.

Because I love the open source community so much, I want to just give this away to you! gh_announce is open source and MIT licensed, you can install it by simply running:

pip install gh_announce

Please let me know if you like this tool, it’ll encourage me to keep writing more such tools in future!

listdir vs scandir vs glob - The one and preferably only way to do it

2019-06-26T00:00:00+00:00

You know, sometimes when I read those python aphorisms like “beautiful is better than ugly” I wonder whether the makers were being sarcastic or real, and I’m not kidding!

Its not just about listdir and scandir, a whole lot of things are ambiguous and you’ll find a lot of different ways to do those same things thus contributing to a non-standard and messy system of working. setuptools and distutils are the obvious examples, there are so many setuptools components that still internally use the old distutils packages that its astonishing. The easy_install is yet another way to do the same packaging dance!

Sometimes, a functionality in shutil is broken and you need to import the alternative from good old distutils which is truly absurd and preposterous. Most recently, I came stumbled across just this:

from distutils.dir_util import copy_tree

The shutil.copy_tree is broken (it doesn’t automatically create non-existent folders and there are some other irregularities), so I had to use the copy_tree from distutils.dir_util module.

Here is another great example. distutils.core, pkg_resources and there is another package called site all doing basically one and the same thing. In this instance, I had to write a local version of pip --freeze that checked my local setup.py and generated the requirements.txt from that instead of globally installed modules. As you can see, there are perhaps a lot of confusing ways of implementing this same thing:

#!/usr/bin/env python
import os
import distutils.core
import pkg_resources

if __name__ == "__main__":
	setup = distutils.core.run_setup("setup.py")
	ss = ""
	for req in setup.install_requires:
		 vv  = pkg_resources.get_distribution(req).version
		 ss += "%s==%s\n" % (req, vv)
	ss = ss.strip()
	print(ss)
	open('requirements.txt','w').write(ss)
	print("Written to requirements.txt")

So, please tell me again, is this for real or sarcasm!

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
**There should be one-- and preferably only one --obvious way to do it.**
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

CodeMirror - A simple and efficient code editor component for your web applications

2019-06-25T00:00:00+00:00

I’m a huge fan of simple things in life, things that achieve a lot with minimal efforts or configuration from the user’s end. And whenever I come across such simple things, I like to share it with everyone and that’s what I’m doing now.

In my recent flask based web project, one requirement was to provide a code editor in the app itself. The web app has a main system and a subsystem, and the user wanted the subsystem part to be dynamically scripted so that she can change that part of the code later and customize it herself. The web-based editor looks something like this (except that it contains the actual code instead of this Hello World placeholder):

This was possible due to CodeMirror which is an open source javascript library, they have a github repository too. You don’t even have to download this library, its available on CDNJS, so you can simply link the stylesheet and two scripts in your html head tag like this:

<script src="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.47.0/codemirror.min.js"></script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.47.0/codemirror.min.css">
<script src="https://cdnjs.cloudflare.com/ajax/libs/codemirror/5.47.0/mode/python/python.min.js"></script>

The first two resources are necessities (codemirror.min.js and codemirror.min.css), whereas the last one for mode (python.min.js) depends on which language or mode you want the editor for. In my case, it was python but there are dozens of modes available for different languages like java, php, ruby, html, css, etc.

The best thing about using this component is getting started itself! All you have to do is create a html textarea (which I already had as a dumb code editor!) and you are good to go:

$(document).ready(function(){
	console.log('adding codeMirror object');
	window.myCodeMirror = CodeMirror.fromTextArea(document.getElementById("txtScript"), {
	   lineNumbers: true,
		mode: 'python',
	});
	window.myCodeMirror.on('change', editor => {
		//console.log(editor.getValue());	
	});
	window.myCodeMirror.on('keydown', editor => {
		//do whatever you want
	});
});

The CodeMirror.fromTextArea() is the important method which directly converts your textarea to a code editor, so simple and so effective! But note that after that, it totally makes your textarea element vanish (display: none) and you’ll have to use the CodeMirror object variable (window.myCodeMirror in this example) to read or write text to it:

var code = window.myCodeMirror.getValue(); //get value from editor
window.myCodeMirror.setValue(code); //set value to editor

This component also has tons of configuration options like tabSize, theme, direction (ltr/rtl), lineNumbers, etc. I hope this editor component will help you if you ever come across a web project that requires it.

Happy coding, code and prosper!

Building a convention for configuration saver and reader module in Python

2019-06-19T00:00:00+00:00

I maintain several python projects on github and some of them like VTScan has a need for user configuration. Now python has a plethora of ways and standards for parsing of configuration files like json, *.ini files, etc., but there is no standard about where to save them on the user’s machine post installation.

One method used by many coders is to save it to the “app path” or the location where your python package itself is installed. On windows machines, this happens to be something like this:

C:\Users\<username>\AppData\Local\Programs\Python\Python36-32\Lib\distutils\<package_name>

This approach has several problems. First, its a location for storing programs and shouldn’t be mixed with data or configuration. Secondly, if your user uninstalls your program in future or even upgrades it (which is a very common scenario), this entire location will simply vanish along with your user’s configuration data! When you upgrade a package like this, pip first removes the old package and then starts to install the new one:

pip install --upgrade package_name

The linux standard way of storing config files for users is ~/.app-name and even ~/.config/app-name where the tilde (~) represents the user’s home directory (which typically expands to /home/username/). This is the way I like the most as its readable and accessible to both apps and humans! There is also the /etc/app-name location on linux but that’s for apps installed with root privileges but as typical python devs, we don’t need to worry about that. If your app requires root, most probably you won’t be using pip/pypi in the first place but choose a proper linux packaging system like apt/dnf/pacman instead.

But storing your configuration files to that location through setup.py upon installation isn’t easy! In fact, you’ll have to import the install class from setuptools.command.install and subclass it to override the post installation process if you wish to go that route!

from setuptools.command.install import install

class PostInstallCommand(install):
	"""Post-installation for installation mode."""
	def run(self):
		install.run(self)
		fpath = os.path.join(self.install_lib, pkg_name)
		fpath = os.path.join(fpath, "cfg.json")
		cfg_dir = os.path.join(os.path.expanduser("~"), ".config/%s" % pkg_name)
		if not os.path.isdir(cfg_dir): os.makedirs(cfg_dir)
		tpath = os.path.join(cfg_dir, "cfg.json")
		shutil.move(fpath, tpath)

However, you can create certain conventions like keeping a standard name (like cfg.json) for all your configurations and storing them to a standard location like ~/.config/app-name as the above code does.

Another thing to take care is that the way to include this configuration file in your setup file is to include it in MANIFEST.in and that thing doesn’t read anything from outside the app’s source directory! You can’t assign it a path like ~/.config/pkg_name/cfg.json, so you’ll have to manually copy cfg.json to your source directory in order to build your package. Then you can set it like this in MANIFEST.in:

include gar_cron/cfg.json

Of course, I can’t automate the whole thing and turn this into a fully “plug and play” library module because things like above are app specific and each dev has to do it for her setup process specifically. But other things I can do like saving and retrieving data from this standard location:

def save(pkg_name, cfgobject, cfgpath=None):
	if cfgpath == None:
		cfgpath = os.path.expanduser("~/.config/%s" % pkg_name)
	if not os.path.isdir(cfgpath):
		os.makedirs(cfgpath)
	cfgpath = os.path.join(cfgpath,  "cfg.json")
	ss = json.dumps(cfgobject)
	open(cfgpath, 'w').write(ss)
	return True


def get(pkg_name , cfgpath=None):
	if cfgpath == None:
		cfgpath = os.path.expanduser("~/.config/%s" % pkg_name)
	if not os.path.isdir(cfgpath):
		os.makedirs(cfgpath)
		return None
	cfgpath = os.path.join(cfgpath,  "cfg.json")
	if not os.path.isfile(cfgpath):
		return None
	ss = open(cfgpath).read()
	return json.loads(ss)

The first function allows you to save your config object (typically a python dict) to this standard location by serializing it to json and writing it to cfg.json. You can also override the cfgpath argument to store it in a non-standard location but you shouldn’t do that unless there is any specific reason.

The second function similarly fetches your config data by reading and deserializing the cfg.json file. Finally, I’ve also written a get_from_cmd() function which is helpful in getting the config directly from the user through command line by passing it a predefined list of config keys:

def get_from_cmd(pkg_name, keys):
	print("Configuration Saver version %s\n" % cfgsaver.__version__)
	obj = {}
	for key in keys:
		try: 
			obj[key] = input("Enter %s: " % key)
		except KeyboardInterrupt as ex:
			return None
	save(pkg_name, obj)
	return obj
	
config_keys = ['github_username', 'alert_email']
config = cfgsaver.get_from_cmd(pkg_name, config_keys)

If you want to use this whole functionality right away, I’ve written a library called cfgsaver which you can install and get started!

pip install cfgsaver

The project README also has special instructions for the specific problems - setting up your MANIFEST.in and customizing the setup.py script. You may also fork or copy this library and create your own version that adapts to your own build process.

Finally, you can also keep using a non-standard method of dealing with config files but that path is riddled with agony and headaches. Besides, if every dev starts following a standard way of saving config files, the world will be so much a better place to live, both for humans and apps!

How to enforce conventional commit messages using git hooks

2019-06-14T00:00:00+00:00

Conventional git commit messages are not just nice to have but great to have. In fact, once you get to know them, you’ll start feeling that they are essential in any serious programming project. Consider the difference between following two commit messages for instance:

git commit -m "added social login feature for authentication using twitter"
git commit -m "feat(authentication): added social login using twitter"

The second one is not only more readable but it also follows standards which makes it easier to integrate with build tools such as Travis CI and Appveyor. Not just that, you can easily automate CHANGELOG generation when your git log looks like this:

> git log --oneline
61c8ca9 (HEAD -> master) fix: navbar not responsive on mobile
479c48b test: prepared test cases for user authentication
a992020 chore: moved to semantic versioning
b818120 fix: button click even handler firing twice
c6e9a97 fix: login page css
dfdc715 feat(authentication): added social login using twitter

All in all, I like conventional commit messages so much that I don’t want to keep it optional but make it the default way of life. How do I do it? The answer is simple: git hooks.

You don’t need to be a git ninja or expert to work with hooks. Suffice it to know that inside your special git repository folder (named .git), there are some other special folders:

logs
hooks
objects
refs

The one we are interested in is hooks. To get the hang of it, create a test project on your machine and initialize an empty git repository by running:

git init .

Now visit the .git folder just generated and navigate to the hooks folder. There will be a bunch of scripts with .sample extension (which means they are all disabled). The one we are interested in is commit-msg. This is the hook that fires just before your commit is made and thus allows you to reject the commit by throwing an error if the message isn’t in proper format.

Create a new script in this directory named commit-msg and add the below code (you’ll need python installed) to it using your favorite editor (mine is notepad++!):

#!/usr/bin/env python
import re, sys, os

def main():
	# example:
	# feat(apikey): added the ability to add api key to configuration
	pattern = r'(build|ci|docs|feat|fix|perf|refactor|style|test|chore|revert)(\([\w\-]+\))?:\s.*'
	filename = sys.argv[1]
	ss = open(filename, 'r').read()
	m = re.match(pattern, ss)
	if m == None: raise Exception("conventional commit validation failed")

if __name__ == "__main__":
	main()

Save the script and make it executable (by running chmod u+x commit-msg on linux, not required on windows). Now head back to your source code folder where you initialized the git repo and create a source file just for testing. Then, git add and try to commit just for testing using a non-conventional message. If all goes well, it should fail like this!

> git commit -m "added a new feature for xyz"
Traceback (most recent call last):
  File "C:/Users/prahlad/Documents/scripts/check_commit.py", line 22, in <module>
	main()
  File "C:/Users/prahlad/Documents/scripts/check_commit.py", line 19, in main
	if m == None: raise Exception("conventional commit validation failed")
Exception: conventional commit validation failed	
Now test with a valid commit message and it should work!

Once you practice this a few times and get a hang over it, you’ll then want to make this behavior default for whatever projects you start using git init in future, isn’t it? Sure, you can do that by creating a global git commit hook template but that will be a post for another day. First things first!

Edit

For those really interested in enforcing and automating this thing, I’ve just published a python package called enforce-git-message. All you need to do is install that python package and it’ll automatically copy the above script to your ~/.git-templates directory and also set the value for git config init.templatedir.

pip install enforce-git-message

References:

Package signing in PIP - It works, in a roundabout sort of way

2019-06-10T00:00:00+00:00

A few days ago, I made this DEV.to post about how Python’s PIP lacks GPG package signing. Well, it turns out that I’m wrong! It does have a package signing process after all, only its one of the most manual, archaic and cumbersome security practices I’ve seen till date.

I came to know about this when I landed on this blog post by a core python developer yesterday. To test package signing in the way described, I created a test package called siterank, a small script to fetch alexa ranking of given websites.

Firstly, its essential that you use only twine to upload a signed package to PyPi because only twine has that feature. Secondly, their documentation seems to be outdated because some arguments don’t seem to work. For example, the --sign argument for specifying signed files explicitly didn’t work for me:

-s, --sign            Sign files to upload using GPG

What worked was to upload the package file and the generated signature file (*.asc) in succession like this:

twine upload siterank-0.2.tar.gz siterank-0.2.tar.gz.asc

Also note that you’ll nowhere see the uploaded signature file on your package page on PyPi. But there are two different ways to verify the signature:

Firstly, you can use the PyPi JSON API. It contains all the uploaded versions in JSON format, notice that in the second package version, the has_sig attribute has been set to true!
The second way is to add the .asc extension to the link to your setup file, in my case it is: https://files.pythonhosted.org/packages/16/f9/1dfce544610b9dcbbfcb4095c8e143c6cfd54b4371ccedc3f73df0a99926/siterank-0.2.tar.gz.asc

So, someone who wants to verify if this package was indeed authored by me can pull this .asc file and match it with my GPG public key (ID E12979BA15FDE7FD - which can be also found by running ` gpg –search-keys prahladyeri@yahoo.com`).

This roundabout way of verification is needless to mention, tedious and cumbersome. This process should be seamless and automated, and included in the pip work-flow itself like apt and dnf have done. The only probable issue is that millions of developers upload their packages to PyPi and everyone may not want to (or lack the knowledge of) signing using GPG keys. So, signing could be kept optional (as it is now) but verification option ought to be there for signed packages as it ensures security and integrity of packages.

Another issue is that of adoption. I’ve noticed from that JSON API that several popular projects like requests, nltk, flask, etc. haven’t signed their packages at all. Its important that more and more developers push signed packages and thus contribute in making PyPi a more secure environment to install and distribute packages.

Security and privacy are perhaps one of the most highly discussed topics of our times. There are attempts by all kinds of people and corporations globally to compromise these by hiding as many things as possible from the plebeians. In light of this, security and privacy should be given the highest priority in open source projects. I hope the Python project understands these concerns and does something about it.

Wordpress to Pelican in 24 hours

2019-05-21T00:00:00+00:00

Today, I finished migration of my blog from a self-hosted Wordpress site to a statically hosted Github Pages site. For the static site generator, instead of choosing Jekyll which is a hot favorite of rubyists, I went for Pelican instead as I figured my Python skills might be somewhat useful in dealing with that.

Having used Jekyll earlier, I felt that Pelican is pretty much the same thing. There is a configuration file in which you define your site parameters, a folder hierarchy for defining your posts and pages, and a bunch of templates (pelican theme) to make some serious customizations.

In this post, I’ll briefly go through the migration process:

Run the standard Wordpress import tool and backup all your posts and comments. It’s better if you disable comments on your site (or maybe even take it down entirely for maintenance until the migration is complete).
Copy the uploads folder containing your images through FTP/SFTP.
Install Python on your machine if not already installed.
Install the Pelican and Markdown packages:
```
 pip install pelican markdown
```
Install additional packages for importing the Wordpress XML:
```
 pip install BeautifulSoup4 lxml
```
Install pandoc as it’s needed for markdown-xml conversion during the Wordpress import. Instructions for installing it for your OS can be found here.
Create a new folder (such as prahladyeri.github.io) for your new blog writing/publishing.

Traverse to that folder through command line and run pelican-quickstart. It’ll ask some basic questions like your blog name, title, etc. You can leave most to their defaults. Once done, it’ll create a folder structure like below:

 d:\source\prahladyeri.github.io
 ├───content                                            
 │   ├───pages                                          
 │   ├───uploads                                        
 │   │   ├───2016
 pelicanconf.py
 tasks.py
 MakeFile

tasks.py and MakeFile are something you can ignore unless you want to automate site generation. Now you can copy the uploads folder brought from Wordpress inside the content folder (it’s the place where your posts will now reside in either markdown (*.md) or reStructured Text (*.rst) formats).
Copy the XML file imported from Wordpress inside the above folder.

Now simply run pelican-import:

pelican-import --wpfile -m markdown <your-file.xml>

The above command converts all your Wordpress posts (not comments) into markdown format and copies it to the content folder. Your old site is now imported to the pelcian system and is ready for generation!
Make some changes to the pelicanconf.py. Set the attributes for AUTHOR and SITENAME. You may also want to set PAGE_URL, PAGE_URL_SAVE_AS, ARTICLE_URL and ARTICLE_URL_SAVE_AS to match your existing site’s URL pattern.
To generate the new site, simply run:
```
pelican content
```
This will generate a folder called output which you can directly serve through github pages! You can also go to the output folder and test it locally by running this command and checking in your browser:
```
python -m http.server
```
From now on, you can compose your new blog posts by simply creating a markdown file (in this case, it’s wordpress-pelican-24-hrs.md, add content to it and simply run pelican content to generate the site! The markdown content should start with header attributes which will tell pelican system about the title, date, category, tags, etc. Here is a brief sample excerpt from this very blog post:

Title: Wordpress to Pelican in 24 hours
Date: 2019-05-21 00:58
Author: Prahlad Yeri
Category: PHP
Tags: Wordpress, PHP, Python
Status: published
Cover: uploads/cover.jpg

Today, I finished migration of my blog from a self-hosted Wordpress site to a statically hosted [Github Pages](https://pages.github.com/) site. For the static site generator, instead of choosing Jekyll which is a hot favorite of rubyists, I went for [Pelican](https://github.com/getpelican/pelican/) instead as I figured my Python skills might be somewhat useful in dealing with that.

This is the very simplest of use cases, of course. For other advanced things like changing the theme templates and CSS, migrating to disqus comments, etc., I’ll make another post later. You may also refer the official pelican docs which has all these details and a lot of other useful information.

Growing intolerance and lack of compassion towards new developers on StackOverflow

2019-05-16T00:00:00+00:00

StackOverflow is the digital equivalent of the Holy See or the Holy Mecca when it comes to programming. It is that place where developers go visit when they don’t have answers to their coding problems in a hope that the Gods of StackOverflow will help them solve it.

But lately, the Gods of StackOverflow seem to be exhibiting a lot of intolerance and impatience towards those who are still learning the craft of programming and new to it. There seems to be a growing belief or consensus that unless you already have a certain level of expertise, you don’t belong here in our chamber.

Take the example of this dude today who had a very simple question about jinja2 templates, a topic related to flask, a python web framework.

{.wp-image-1132 .size-large width=”1024” height=”458”}

A question about jinja2 abruptly closed as a duplicate

I was about to answer this question when it was abruptly closed for being a duplicate. The person who closed this didn’t even consider that a beginner coder (considering the reputation score) may not even know what the capabilities of jinja2 were. How the heck was s/he supposed to know that the answer to this question was to be found outside the context of jinja2? Everyone on this planet isn’t a born expert in web development. The coder expected a solution in the context of jinja2 templates itself and the right way would have been to guide him/her and explain that this isn’t possible through jinja2 but front-end scripting is required here. Then the answering post could proceed to provide a few helpful example links from StackOverflow or someplace else.

I want to ask those intolerant ones: consider that you yourself could have needed this hand-holding and guidance when you were beginning to code some years earlier? Just because coding has become so easy and second nature to you now, it doesn’t give you any right to behave with arrogance. Knowledge needs to be used with grace and caution, with power comes the responsibility of using that power.

This kind of behavior is quite common from what I’ve been reading lately on social media too and my observation today just confirms it. I hope that more and more senior developers develop compassion towards others and treat this site as the temple of knowledge which it really is, not an egotistical chamber to shut down other programmers.

A dark day for consumer rights: Right to repair bill killed in Canada thanks to corporate lobbying

2019-05-03T00:00:00+00:00

When Linus Torvalds wrote the Linux Kernel in 1991 or Ian Murdock first released the Debian OS in 1993, talking about the “openness” of a PC or the “right to repair it” would have been a laughable affair! 8086 was an open standard then (just as standards should be) and anybody was free to write an operating system for it (such as Linux), or user land utilities (such as GNU) or even define a desktop standard (such as Xorg).

But in 2019, that’s no longer the case with standards. Looks like humanity is going backward in terms of progress as standards are becoming more and more closed and proprietary instead of becoming open and transparent. Companies like Apple are successful in not only creating entire dependencies of such closed standards but are fighting tooth and nail so that their closed standards get a stamp from the legislature too.

The law makers of Ontario, Canada succumbed to this massive corporate lobbying today and killed the “right to repair” bill before it could even see the lights of the day. This bill was asking for a very basic thing - the right to open your own device (which you’ve already paid for) and fix it for any problems. In order to empower a user to do this, the bill called for forcing tech companies like Apple to provide the following at a fair price:

Repair Manual
Official Parts
Diagnostic Tools

This is something very basic, something the Stallmans and Torvalds of the world have already been doing since 90s. Canada sent a very wrong signal to the tech world today, this would have been the first such pro-consumer law in the world, if passed. About 20 American states are also considering a similar right to repair bill and this sets a very wrong precedent to those decision makers. Both USA and Canada are highly developed countries and the whole world looks up to them for framing their own tech laws, so this is a very bad thing that happened today. I hope the American states will act sanely and pass this bill otherwise any hope left for open standards and protocols for the future generations will be lost. Since innovation blossoms in openness and gets stifled in closed environments, this will sound a death knell to innovation too.

It is now left to organizations like EFF and internet freedom activists like Richard Stallman to protest this decision and ensure some sanity in the tech world which is getting increasingly crony capitalistic.

Foxy firefighters and the clash between individual and collective liberalism

2019-02-02T00:00:00+00:00

The biggest speculation in the tech industry these days concern net neutrality and the last week was no different. Whilst I’m happy that TRAI is clear on upholding the net neutrality in my own country, India, the clouds of doom could be clearly seen in USA as the FCC is trying hard to banish net neutrality and bring control of the Internet in the hands of a few large companies.

This is quite worrying for Internet in general because the world looks up to the United States for laws governing the digital space, not India. What’s hilarious is that the FCC is resorting to unbelievable arguments like broadband isn’t a telecommunication service in their desperate attempt to repeal net neutrality. I hope the US legal system is smart enough to discard such illogical arguments and the light of justice prevails in the country where Internet was first invented.

{.size-medium .wp-image-1087 .aligncenter width=”300” height=”200”}

The positive thing though is that those fighting on the side of net neutrality are also strong in numbers and include companies like Reddit, Mozilla, Vimeo, etc. along with 22 state attorneys. Also, unbelievably, in the middle of this protest against repeal of net neutrality are a bunch of fire fighters who also want net neutrality to prevail! Fire fighters of California are citing public safety as a reason to uphold net neutrality and logically so. According to the linked ABC news report, California witnessed the largest fire in its entire history and the firefighters encountered an unexpected danger - they couldn’t communicate with their team members as Verizon was deliberately throttling and slowing down Internet speed on their phones. This is just a small sample of what could happen across the world if net neutrality was repealed for good.

In other news, Facebook seems to be offering teenagers 20/month in exchange for their phone activity data. While privacy advocates are all pitchforks on this happening and calling for Facebook to be made liable, this has a very large implication on the ever ongoing debate on “Individual Liberty vs Collective Liberty”. For me, personally, it makes sense that digital privacy is a very important thing and online companies shouldn’t be encouraged by allowing to run schemes like these when studies have shown that they already collect a whole lot of user data by default (location tracking, analytics, diagnostic data, etc.). Any other ration dude who has some knowledge of Information Technology will also think similarly, I suppose. However, if I try to advocate this knowledge to others and call for laws preventing such individual sale of private data, we are venturing into the area of collective liberalism, and here comes the danger of a clash happening between collective and individual liberalism. The thing is that those teenagers (however misguided they are), have full liberty to trade their private data with facebook or whatever social media company they want, and preventing them from doing that might be against the ethics of individual liberalism, that’s just my few cents. In all probability, this debate could continue for years to come and so will the privacy debate, hopefully a solution will be found which is favorable and acceptable to everyone.

Last but not the least, there are many studies like these sprouting up these days claiming that deleting Facebook or Twitter accounts bring happiness in people’s lives. However, I want someone to perform another study into the mystery of why do so many people decidedly stay unhappy by constantly plugging into the social media sites!

Docker trick: running containers efficiently using a bash script

2018-09-14T00:00:00+00:00

One of the basic problems with running a docker image is that its too easy to spew up multiple instances or containers of the same image. Consider running the following container for instance:

docker run -it -e AWS_ACCESS_KEY_ID="test" -e AWS_SECRET_ACCESS_KEY="test" \
--net=host "prahladyeri/testimage:latest" /bin/bash

This will start a fresh container instance from image prahladyeri/testimage:latest and you may optionally pass environment variables (using -e switch), and the /bin/bash argument to manually trigger a command inside the container instead of running the default one. However, once you exit the container, you don’t have any control over that container instance. Running the same “docker run” command as above is just going to spew another container instance for the same image, while the old one still stays in memory. Repeating this process multiple times is a wastage of invaluable memory and cores which isn’t the goal of using docker at all!

{.alignnone .size-medium .wp-image-1073 width=”300” height=”232”}

A more efficient way of handling things is to check the existence of a container instance already running for the image, and restart that one if required:

docker ps -a -f "ancestor=prahladyeri/testimage:latest" -q -l
f7261ea890ce

The above command returns the id of an already running container (if any), so using that you can just restart that same container without passing all the parameters again as follows:

docker start -i f7261ea890ce

Of course, you can automate this whole process by writing a single bash script called start_container:

#!/bin/bash
imgname="prahladyeri/testimage:latest"
imgid=$(docker ps -a -f "ancestor=$imgname" -q -l)

if [ -z "$imgid" ]; then
  echo "creating new instance for $imgname"
  docker run -it -e AWS_ACCESS_KEY_ID="test" -e AWS_SECRET_ACCESS_KEY="test" --net=host "$imgname" /bin/bash
else
  echo "instance found"
  docker start -i $(docker ps -a -f "ancestor=$imgname" -q -l)
fi

Just replace the $imgname variable with that of your own image name, and pass any arguments you want (for example, -e AWS_KEY_ID) to the container on line 7. This script will start a new container only if none are found already, otherwise, it will just restart an existing one!

Hope you find this useful.

Performing a little usability tweak on the XFCE window recycler in greybird theme

2018-09-11T00:00:00+00:00

Xubuntu is my favorite distro, hands down and the default Greybird theme is just wonderful! There used to be a time when I liked Ubuntu-MATE too, but not so much since they took the road to GTK+3! Coming back to the topic, XFCE works great but it has a small usability issue in the default Greybird theme which annoys most power users.

ALT+TAB is the usual way to recycle window icons on any DE, and the “tab” key determines the currently focused window. Now, one of the Greybird theme developers decided in his profound wisdom that a mouse move over the cycling dialog can cause a window focus too. Since the mouse cursor occupies a central position most of the time, it automatically causes an item in the cycling dialog to be selected which is not intended at all! While this could be a great “feature” for those users who like juggling with their mouse while pressing the ALT+TAB keys, I hardly think any power user would like to do that. For most users, this could be easily a cause of annoyance or frustration since they have to move the mouse out of the dialog’s range to ensure that it doesn’t interfere with their selection again.

As you can see, the blue background represents the selected/active window. Now, imagine intending to select the firefox window by pressing ALT+TAB, but terminal window gets selected instead since the mouse happens to be placed there! Astonishingly, however, I couldn’t find any discussions or issues raised regarding this anywhere on the internet except this one thread on XFCE forums. After making some tweaks to their code, I was able to come up with my own version which resolves this issue:

include "/usr/share/themes/Greybird/gtk-2.0/gtkrc"

style "xfwm-tabwin-tweak"
{
 Xfwm4TabwinWidget::border-width = 1
 Xfwm4TabwinWidget::border-alpha = 0.9
 Xfwm4TabwinWidget::icon-size = 64
 Xfwm4TabwinWidget::listview-icon-size = 16
 Xfwm4TabwinWidget::preview-size = 512
 Xfwm4TabwinWidget::alpha = 0.9 #0.8
 Xfwm4TabwinWidget::border-radius = 5 #10

bg[NORMAL] = shade (0.45, @bg_color_dark) # widget background color
 bg[ACTIVE] = shade (0.65, @selected_bg_color)# when keyboard and mouse focus on the same item
 bg[PRELIGHT] = shade (0.45, @bg_color_dark) # color of item with mouse hovering on it, which we want to make it
 bg[SELECTED] = shade (0.65, @selected_bg_color) #color of selected item using keyboard

fg[NORMAL] = shade (0.8, "#fff") #shade (0.8, @base_color)
 fg[ACTIVE] = "#fff" #@base_color # text color of item where our mouse and keyboard meet
 fg[PRELIGHT] = shade (0.8, "#fff") #shade (0.8, @base_color)
 fg[SELECTED] = "#fff" #@base_color

engine "murrine" {
 roundness = 6
 }
}

widget "xfwm4-tabwin*" style "xfwm-tabwin-tweak"

Save the above file as ~/.gtkrc-2.0 and change your theme from Greybird to something else and back again, and this issue will be resolved.

Linux DIY: Creating a multiboot USB drive with ISO images of multiple distros

2018-09-11T00:00:00+00:00

Though I’m not a distro-hopper exactly, I keep trying live versions of popular linux distros like debian and ubuntu every now and then, and while I generally use the “dd” command which works right out of the box, today I thought that instead of burning a new ISO image each time, why not create a single 16GB USB stick which can boot multiple distro images!

dd if=xubuntu-18.04.1-desktop-amd64.iso of=/dev/sdb bs=1M

This simple dd command does the job, but you are left with a single ISO image occupying the entire disk space which is an inefficiency if you have a larger drive that can hold multiple images. So, let’s learn how to reclaim all that space by creating a single USB which can boot multiple ISO images. Though the process is a bit more involved than the dd command, the reward is obviously worth it too.

Step-1: Format your USB drive to FAT32, there should be a single partition on the drive with boot flag enabled. - Also, label the drive as MULTIBOOT or something for recognizing it easier. Though the linked article below uses the fdisk/mkfs.vfat commands, you can even use graphical tools like gparted to make this process simpler.

{.alignnone .size-full .wp-image-1020 width=”774” height=”520”}

Step-2: Install grub2 on the newly formatted drive, create a boilerplate grub.cfg file:

Close the gparted window if its open, otherwise the drive won’t be mounted.
Type sudo grub-install --force --no-floppy --boot-directory=/media/prahlad/MULTIBOOT/boot /dev/sdx (replacing path with yours, x with your actual USB device)
Type cd /media/prahlad/MULTIBOOT/boot/grub (or whatever your path is)
Type wget http://pendrivelinux.com/downloads/multibootlinux/grub.cfg (to get the grub.cfg file)

Step-3: Copy whatever ISOs you want into the root folder of your drive. For the purposes of this example, let’s consider latest LTS releases of Ubuntu and Xubuntu (swap with any other flavors or debian too if you prefer):

cp ubuntu-18.04.1-desktop-amd64.iso /media/prahlad/MULTIBOOT/
cp xubuntu-18.04.1-desktop-amd64.iso /media/prahlad/MULTIBOOT/

Step-4: Edit the grub.cfg you just downloaded, and configure entries for your ISOs accordingly. Its important that you do this carefully:

# This grub.cfg file was created by Lance http://www.pendrivelinux.com
# Suggested Entries and the suggestor, if available, will also be noted.

set timeout=10
set default=0

menuentry "Ubuntu Desktop ISO" {
 loopback loop /ubuntu-18.04.1-desktop-amd64.iso
 linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=/ubuntu-18.04.1-desktop-amd64.iso noeject noprompt splash --
 initrd (loop)/casper/initrd.lz
}

menuentry "Xubuntu Desktop ISO" {
 loopback loop /xubuntu-18.04.1-desktop-amd64.iso
 linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=/xubuntu-18.04.1-desktop-amd64.iso noeject noprompt splash --
 initrd (loop)/casper/initrd
}

As you can see, I’ve changed only the menu entries to keep things simple. The arguments are also self apparent. For instance, loopback and loop attributes are used because we aren’t booting directly from the USB, but mounting the virtual loop device (ISO file) instead.

However, note that there could be subtle differences between distros, xubuntu uses /casper/initrd instead of /casper/initrd.lz, for instance. To know the right entry for your own distro, compare the “/boot/grub/grub.cfg” file of your distro ISO to get a hint about what menu entry to add. If your ISO isn’t one of the ‘buntus or a debian, then it might get trickier.

Step-5: Boot from your USB drive and ensure that it works.

Make sure that you boot from your newly created USB drive, not from the hard drive as usual. If everything goes right, you should be able to see a grub boot screen with the two menus for Ubuntu and Xubuntu.

Once this process is successful, adding new ISOs to your drive is only too easy. Just copy the brand new ISO to your drive’s root directory and just add the necessary entry in /boot/grub/grub.cfg, that’s it!

Finally, you’ll have to enable legacy boot option if not already done in your bios settings, this process won’t work with EFI/UEFI systems since the drive doesn’t have an EFI partition. I might do another post about creating a similar drive for EFI/UEFI setups in future, until then you can refer to this reddit post by /u/need2burn.

Best of luck!

Reference:

http://www.pendrivelinux.com/boot-multiple-iso-from-usb-via-grub2-using-linux/

Windows recipe: How to quickly track your active network connections using resmon utility

2018-08-28T00:00:00+00:00

Windows 10 is a great operating system when its clean and well-maintained but once you start cluttering it with more and more apps, your Internet speed typically starts to slow down and it becomes increasingly difficult to find out which background app is consuming your bandwidth.

One way of tackling this is to harden the windows firewall by restricting outgoing TCP connections too (incoming are already restricted by default). But this is quite cumbersome in the long run as you will have to whitelist each and every executable that needs to access the internet which can be a difficult and time consuming task.

Another quick way of finding out which app is consuming your most bandwidth (whether on the background or foreground) is to fire up the Windows Resource Monitor by running resmon.exe which is a nice Windows utility program that comes by default. By clicking on the Network tab and expanding the Network Activity pane, you’ll come to know exactly how much bandwidth each executable is eating on your system!

Resource Monitor (resmon.exe) running on Windows

Once you know the culprit EXE causing the bandwidth drain, you can either uninstall that app or block it in the windows firewall. All in all, resmon.exe is a great utility in the hands of the Windows power user who wants to take control of his/her operating system. I hope you find this utility useful.

Userscript recipe: Adding auto-refresh button to gmail classic

2018-08-28T00:00:00+00:00

Call me old, but I much prefer the older classic GMail interface to the modern bloated one. Not only is the classic version less bandwidth consuming, but also simplistic in nature with just the buttons and tools that we need.

I somehow don’t like it when zillions of AJAX scripts do all kinds of magic in the background of the web-page. However, I still want the GMail tab to auto-refresh itself and check new mails while I’m working in another tab in my browser. If only Google provided a way to auto-refresh in the classic interface! Since there isn’t one, I wrote a user-script to add this functionality:

{.size-full .wp-image-1009 width=”1171” height=”561”}

GMail Classic

You can use any Script extension for Chrome/Firefox such as GreaseMonkey or TamperMonkey to add this userscript to your browser. I really hope you find this userscript useful!

Beware of Uber's vomit fraud when booking a cab next time

2018-07-24T00:00:00+00:00

According to some recent reports by top technology blogs, Uber drivers have been reportedly engaging in an activity called “vomit fraud”. Basically, Uber (and many other cab services too) allow their drivers to collect a fine of up to $150 if their customer creates a mess in the cab (such as puking).

However, we aren’t talking about the occasional real puke or vomit here, drivers are reportedly exploiting this feature to fraudulently claim a “cleaning fee” even when no such thing was done by a customer.

{.alignnone .wp-image-994 .size-medium width=”300” height=”200”}

For instance, according to this Gizmodo report a Miami resident was recently fined twice fraudulently for this. And whilst you might recover this fine by engaging with their customer care and maybe exchanging several emails with them, the process is almost infinitely tedious and there isn’t a guarantee that you may get your money back.

To make things worse, as per this Mashable report, a tech savvy driver may also photoshop a fake picture of a puke and provide it as evidence thus making your chances of recovering the fine almost zero.

The practice has become so rampant that the drivers are openly admitting doing this on social media platforms and treating it casually. From the linked Mashable report:

>> “lol vomit fraud is the oldest trick in my book,” one Redditor — with a history of ride-hail related posts — wrote in r/uberdrivers. “I’ve successfully done it many times when [passengers] are rude, slam my doors, or make me wait too long. Makes them learn the hard way.”

I don’t know what could be the solution to this other than some basic “street-smartness” by customers and maybe the built-in review system. If drivers who indulge in such fraudulent behavior are reported frequently and called upon by customers in their reviews, that should ideally purge the system of bad drivers in the long-term.

XFCE panel indicator plugin background fix

2018-07-21T00:00:00+00:00

The indicator plugin in XFCE panel has been a very useful but also a very controversial plugin! The reason is that its the only plugin on the XFCE panel developed in GTK3 (whereas rest of the panel is developed in GTK2) and this causes some theming issues.

Specifically, only those themes will work for the panel who’s developers have taken care to style both the GTK2 and GTK3 components, otherwise it results in an odd background color around the indicator plugin which isn’t in sync with rest of the theme:

{.size-full .wp-image-976 width=”380” height=”130”}

XFCE indicator-plugin background issue

Bug #1408979 has been reported on launchpad for this and even askubuntu is filled with a bunch of questions pertaining to this issue. I came across this while trying to port ambiance looks to my xubuntu desktop.

The fix is really simple if your theme already has GTK3 support (but not for the indicator plugin specifically). GTK3 support typically exists for your theme if it has /gtk-3.0 folder in its root folder. All you have to do is open the /<your_theme>/gtk-3.0/gtk.css and add this style patch to it:

/*xfce panel workaround */

.xfce4-panel {
  background-color: @dark_bg_color; 
  font: normal;
}

.xfce4-panel .button {
  background-image: none;
  background-color: @dark_bg_color;
  border-radius: 0;
}

.xfce4-panel .button:active,
.xfce4-panel .button:checked {
  background-image: none;
  background-color: shade(@dark_bg_color, 1.20);
  border: none;
}

.xfce4-panel .button:hover,
.xfce4-panel .button:active:hover,
.xfce4-panel .button:checked:hover {
  background-color: shade(@dark_bg_color, 1.20);
  border: none;
}

Make sure to replace the \@dark_bg_color with whatever your theme’s background color is. After the above change, just apply your theme using the theme switcher and you will see this problem magically fixed!

Now in cases where your theme doesn’t have GTK3 support, it might get tricky. I faced this same issue with the Ambiance XFCE theme, for instance. What I did as a workaround was copy the gtk-3.0 folder from the official Ambiance theme from ubuntu-mate-themes package to the linked XFCE theme’s folder and then did the above fix. Of course, I could have used the official theme itself in the first place, but it doesn’t support xfwm4.

I hope the above information helps you if you were looking for a fix to this exact same issue.

How to make your Xubuntu 16.04 desktop look and behave like Ubuntu-MATE

2018-07-18T00:00:00+00:00

Xubuntu is one of the most popular among the “lighter” distros as it hardly consumes 200MB when idle and even older machines are able to run with acceptable performance. Ubuntu-MATE which is based on MATE Desktop (a GNOME-2 fork) is also a similar competing distro though its debatable whether its as light as Xubuntu in terms of resource consumption.

Xubuntu Desktop which looks like Ubuntu-MATE

Making a switch from Xubuntu to Ubuntu-MATE totally makes sense if you have any particular reason for it, but I’ve observed that many people make that switch only because they like the astonishing looks of Ubuntu-MATE which are admittedly and relatively better compared to XFCE. For whatever reason (such as familiarity) if you want to just stay on Xubuntu but still want to have the “look and feel” of Ubuntu-MATE, its very easy to achieve that, you just have to follow this brief process described in this article.

Step-1: Download the Ambiance theme for XFCE from here and extract them to your ~/.themes/ folder.

Whilst the official Ubuntu packages mate-themes and ubuntu-mate-themes also come with a version of Ambiance theme, the window title background didn’t turn dark when I applied it for some reason. You can still have those packages installed if you want, but the one in your ~/.themes/ folder will override any installed themes.

Step-2: Apply Ambiance GTK theme.

Head over to Appearance applet in your XFCE Settings Manager and switch the GTK theme to Ambiance which you should now see after performing step-1:

Settings Manager->Appearance

Step-3: Apply Ambiance Window Manager theme.

The next step is to do the same for Window Manager theme. Head over to Window Manager applet and make the switch there too:

Settings Manager->Window Manager

Step-4: Install Ubuntu-MATE icons.

This is for adding those green MATE icons. Just run this command to install the official package:

sudo apt install ubuntu-mate-icon-themes

Again open the Appearance applet but this time click the Icons tab on the top, and just select Ambiance-MATE from there:

Settings Manager->Appearance (Icons Tab)

Optionally, you can also click the Fonts tab and change the font to Ubuntu or one of its derivatives to get that Ubuntu-MATE feel!

Step-5: Use the magic switch: XFCE Panel Switch.

The XFCE Panel Switch is a wonderful tool that drastically alters the layout of your XFCE Desktop, just search for “XFCE Panel Switch” in your whisker menu or find it under Settings in your regular menu. Start that tool and you’ll be able to switch to any desired desktop layout (choose “GNOME 2” for the looks of Ubuntu-MATE and click “Apply” icon which is the first in the bottom list of small icons):

XFCE Panel Switch

Once you do that, you should be able to get a perfect Ubuntu-MATE look alike such as the one shown on the top screen. Finally and optionally, you can also install the official package ubuntu-mate-wallpapers if you like their wallpapers too.

Backbone.Events+Promises+async/await is a great combination for building javascript apps

2018-07-12T00:00:00+00:00

At the risk of being a contrarian, I’d like to show in this article how exactly can the Backbone’s Events model be combined with the more modern constructs of Promises and async/await to build a killer app using JavaScript.

First, let’s try to understand why do we need these constructs for asynchronous programming when JavaScript itself is a mostly asynchronous language featuring events and function callbacks. The Promise API was introduced in ES6 standard as a way of preventing callback hell:

function someFunction(function anotherFunction(data){
  function yetAnotherFunction() {
    function reallyCoreFunction() {
      //readability sucks now!
    }
  }
});

This is only three layers of callbacks, imagine what will happen when there are a hundred of them which is quite possible in a medium sized web app with multiple ajax calls and a dynamic interface. To solve this problem, functions started returning a “Promise” object instead of a callback function, so instead of multiple layers of callbacks, we can now use “function chaining” using the then keyword like this:

someFunction()
.then(function(){
  yetAnotherFunction();
})
.then(function(){
  reallyCoreFunction();
})
//readability is better now!

As you can see, the Promise API has improved the readability to a considerable extent. Instead of a callback mechanism, the callee returns an object called Promise which can be chained for further execution. For this to happen, the callee has to call resolve() in order for the caller to end the wait and trigger further execution in the next chain:

{.size-full .wp-image-921 width=”755” height=”222”}

Promise Model

But the async/await construct goes even further than this. You don’t even need to do function chaining using then, but the await keyword itself is enough to do this. Internally, the async/await model uses the Promise API to achieve its end because even in this model, the callee has to make the resolve() call in order to return execution control to the caller and the process continues after the next await statement:

{.size-full .wp-image-922 width=”755” height=”222”}

async await Model

However, one disadvantage of async/await is that the caller can only wait, it cannot receive a returned value which is possible in the Promise model. Another disadvantage is that the caller function itself who invoked the await statement needs to be declared as async which means it cannot be a part of a sequential statements in another process, but has to work independently as shown in this example:

var sequentialStart = async function() {
  console.log('==SEQUENTIAL START==');
  const slow = await resolveAfter2Seconds(); // If the value of the expression following the await operator is not a Promise, it's converted to a resolved Promise.
  const fast = await resolveAfter1Second();
  console.log(slow);
  console.log(fast);
}

Finally, the most important thing here is to realize that both Promises and async/await are just one level of asynchrony, let’s call this detail asynchrony or micro asynchrony, but there is also the other kind that works at the broader level which can be termed macro asynchrony, and this is where Backbone.Events comes into the picture.

Consider that you have a complex web application with several views and each component should be able to trigger a message or event to any other component asynchronously in order for the app to render and function properly. Consider an app with a loginView that needs to trigger an alert that a user has just signed in. Now, keeping the separation of concerns, the best practice here would be that the loginView shouldn’t try to render that part of the DOM and leave it to the other component: navbarView. Now, how can our loginView be able to “tell” the navbarView asynchronously over the wire to display that alert? Both Promises and async/await are of no use here, and hence we need a macro-level all-purpose Event Bus, similar to the one that Backbone.js provides us:

{.size-full .wp-image-920 width=”757” height=”278”}

backbone.events Model

In this case, our caller and callee are inside separate components and don’t even know each other. And separation of concerns imply that they cannot even contact each other directly. Now, Backbone provides this useful trigger/listen mechanism called Backbone.Events using which any object in your app can listen to the events triggered on any other object! In this case, the loginView triggers an event called “navbarView.alert” and thus “tells” the navbarView that it has to do something (in this case show an alert). This mechanism also supports passing of any kind of data along with the triggering of event (in this case the message parameter).

Thus, we can see that combining the power of all these asynchronous models, we can easily handle any level of complexity in our app, provided that our app itself is structured in the right way from the start.

WordPress DIY: Adding syntax highlighting to your WordPress blog without using an external plugin

2018-07-10T00:00:00+00:00

Just as my other articles in WordPress DIY series, this one also focuses on doing everything yourself by writing the code rather than using any external dependencies. There are two popular open source implementations of Syntax Highlighting JavaScript libraries: Google’s Prettify and Alex Gorbatchev’s Syntax Highlighter, and in this article, we will use the former. Writing your own plugin for syntax highlighting is very straightforward if you know what you are doing. Just create a file named /wp-content/plugins/wp-prettify.php in your WordPress installation and add the below code to it:

<?php
/**
 * @package WP Prettify
 * @version 0.1.0
 */
/*
Plugin Name: WP Prettify
Plugin URI: http://github.com/prahladyeri/wp-prettify
Description: Wordpress implementation of Google Prettify Syntax Highlighter
Author: Prahlad Yeri
Author URI: https://prahladyeri.github.io/
Version: 0.1.0
License: GPL version 2 or later - http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
*/


function add_syntax_highlighter() {
$text=<<<EOD
<script>
document.addEventListener('DOMContentLoaded', function(){
    var elements = document.getElementsByTagName('pre');
    for(var i=0;i<elements.length;i++) {
        if (elements[i].className.indexOf("prettyprint") == -1) {
            elements[i].className =   elements[i].className + " prettyprint "
            console.log('adding');
        }
        else {
          console.log('class already exists');
        }
    }
});
</script>
<style>
pre.prettyprint.prettyprinted {
  border: 0px;
  padding: 5px;
  font-size: 15px;
}
</style>

<script src="https://cdn.rawgit.com/google/code-prettify/master/loader/run_prettify.js"></script>
EOD;

    echo $text;
}

add_action('wp_head', 'add_syntax_highlighter');
?>

Remember to enable the plugin by navigating to /wp-admin before using it.

This function first adds a prettyprint class to all your <pre> tags to tell the Prettify library to highlight this block and thus making syntax highlighting painless on your part. The advantage of using Google’s solution is that it automatically detects the programming or scripting language inside the block, and you don’t have to use additional tag markup like “lang=php” or something like that.

The addition of CSS <style> tag in the function is optional and not really needed. Its just my preference to hide the border and increase the padding and font size a bit.

The demonstration of this self-written plugin is quite evident as this very site runs on this plugin and the above code block is highlighted using the Google’s Prettify library. If you can use the Chrome or Firefox Developer tools to analyze that <pre> block, it will show you that a prettyprint class has been added!

So, go ahead and write this plugin if you are building a WordPress blog for yourself and want to do this in a DIY way.

Best minimalist social networks for users tired of clutter

2018-07-10T00:00:00+00:00

The internet is a vast place, brimming with content that ranges from engaging and informative to downright overwhelming. For those who find comfort in clean, text-rich experiences, it can be a challenge to sift through the digital noise of flashy animations, autoplaying videos, and invasive ads. Minimalist social networks cater to users who prefer simplicity, efficiency, and thoughtful engagement over the sensory overload common to mainstream platforms. If you’re a fan of the classic Reddit or similar text-centric sites but are disillusioned with their current trends, this guide is for you. We’ll dive into the best alternatives and what makes each worth a look.

1. Why minimalist platforms matter

Minimalism on the web isn’t just an aesthetic choice; it’s a functional necessity for many users. Sites like Facebook, Twitter, and even new Reddit versions have increasingly incorporated dynamic elements that can hinder user experience. For those with slower internet connections or those who prioritize productivity, minimalist platforms offer:

Faster load times due to reduced multimedia and animations.
Simpler user interfaces that help keep the focus on content.
Improved reading and navigation without visual clutter.
Reduced cognitive load for more focused interactions.

Here’s a roundup of some of the best minimalist social networks available:

Hacker News

Niche: Technology and startups
Key Features: Plain text format, high-quality discussions, strict moderation
Pros: Excellent for tech enthusiasts, devoid of visual distractions.
Cons: Limited to tech and startup news, may not appeal to a general audience.
Website: Hacker News

What sets it apart: Hacker News was created by Y Combinator and has maintained its no-frills interface since inception. Its orange-and-white theme, simple text links, and threaded comments are loved by developers, entrepreneurs, and tech enthusiasts for their straightforwardness and depth.

Hubski

Niche: General interest, thoughtful discussions
Key Features: Community-driven, tag-based content, user-focused interactions
Pros: Emphasizes quality over quantity, strong community feel.
Cons: Smaller user base, may feel quiet compared to larger platforms.
Website: Hubski

User insights: Hubski’s community is known for its civility and depth. Conversations often venture into philosophy, literature, and less-mainstream topics, making it a haven for readers looking to escape low-effort posts and clickbait headlines.

Mastodon

Niche: General social networking with a twist
Key Features: Decentralized, federated servers, open-source
Pros: No single entity controls it, greater privacy options.
Cons: Finding the right server (instance) can be tricky at first.
Website: Mastodon

Why it’s unique: Mastodon allows users to join different “instances” tailored to their interests. Each instance operates like its own social network but is federated with others, allowing cross-communication. This structure supports a diversity of communities without the top-down control of traditional networks.

Snapzu

Niche: Broad topics, storytelling and discussions
Key Features: Tribe-based topics, invite-only access
Pros: Polished interface that is still minimalist, respectful discussions.
Cons: Smaller community size, invite requirement.
Website: Snapzu

Inside Snapzu: Snapzu’s “tribes” are akin to subreddits but with a cleaner, more user-friendly design. Discussions here lean toward thought-provoking and detailed, offering a breath of fresh air to those who miss old-school forums.

3. Comparison table for minimalist platforms

Platform	Niche	Key Features	Pros	Cons
Hacker News	Technology/startups	Plain text, quality discussions	Fast, simple UI, high-quality content	Niche-focused
Hubski	General interest	Tag-based, thoughtful posts	Community-driven, rich discussions	Smaller user base
Mastodon	General social networking	Decentralized, federated	Privacy-focused, varied content	Steep learning curve for new users
Snapzu	Broad topics	Tribe categories, minimal UI	Polished design, respectful community	Invite-only access
Discuit.net	Community/social sharing	Markdown support, simple interface	Lightweight, ad-free, active communities	Limited user base, less feature-rich
Peerlist.io	Tech professionals	Profiles, project showcase	Great for networking, verified credentials	Smaller network compared to LinkedIn
Lemmy	General/community forums	Federated, decentralized	Privacy-focused, Fediverse support	Relatively new, limited user base
kbin	Aggregator and microblogging	Federation, ActivityPub support	Decentralized, ad-free, privacy-focused	Limited reach, niche audience
Steemit	Content creation	Blockchain-based rewards	Incentivizes content creation	Cryptocurrency volatility
SaidIt.net	Social news	Voting, free speech	Privacy-focused, open community	Light moderation, niche audience

4. Honorable mentions

4chan
Known for its image board structure, 4chan remains an anonymous, minimalist site that caters to all types of discussions, from memes to activism. It has a reputation for being chaotic, so users should proceed with caution.

Medium and Quora
While Medium and Quora are not strictly minimalist, they deserve mention for their cleaner interfaces compared to mainstream social networks. Medium focuses on quality storytelling and articles, while Quora is home to user-generated questions and answers across a wide range of topics.

Minimalist platforms remind us that social media doesn’t need to be a never-ending scroll of autoplay videos, pop-ups, and ads. They offer the opportunity to:

Reconnect with genuine discussions and well-thought-out content.
Enjoy a seamless, fast experience without overwhelming multimedia.
Join niche communities with shared interests for meaningful interaction.

In a world where many platforms chase after engagement metrics, minimalist social networks stand as testaments to the power of simplicity, encouraging users to slow down, read, and think.

How to create a Server-less Google Drive client using only HTML and JavaScript

2018-07-09T00:00:00+00:00

A few days ago, I had to work on a project of this kind and the information and documentation available on this topic was quite bewildering. As a result, I decided to write this article in order to make everything available at one place.

Almost everyone knows about the Google API Python Client(and similar others in Java, PHP, etc.) which can be accessed from the server, but few are aware that Google even supports a totally server-less implementation of a client using only client-side technology (HTML/JavaScript). The only caveat here is that your app won’t get a permanent access token like the server side apps do, but instead, has a temporary access until your web page isn’t closed. Here is how to go about doing it:

Step-1: Register your App by visiting the Google Cloud Console:

As usual, visit the Google Cloud Consoleand register your app first, enable the Google Drive API by visiting “APIs & Services”=>“Dashboard”, then click on “Credentials” to create an API Key, an OAuth credential and setup the authentication screen for your user. While setting credentials, remember to add your app’s URL in authorized Redirect-URIs section. If you have multiple URLs for your app (like localhost for testing, www.YourSite.com for production, etc., then add them all to the list):

Credentials Configuration: Redirect URIs

This is what your user will see when they visit your app and it redirects them to Google for getting permissions for Drive access. Click on the “OAuth consent screen” tab and configure it with your app’s name, logo, etc.

Step-3: Add script references.

After you create the credentials and get your ClientID, API Key and secret key (not useful for our app, really), the next step is to start building the app. I usually prefer to keep the JavaScript logic in a separate file such as app.js instead of keeping it in the main index.html. Just add a script tag in your main index.html and add a reference to this app.js in which you’ll write the Google Drive logic:

<script src="/static/app.js"></script>

I’ve kept the app.js in a sub-folder named /static, if yours is different, then adjust the path accordingly. You’ll also need to add a reference to Google authentication library in order to perform the OAuth authentication:

<script async defer src="https://apis.google.com/js/api.js" 
 onload="this.onload=function(){};handleClientLoad()" 
 onreadystatechange="if (this.readyState === 'complete') this.onload()"></script>

Finally, also add a reference to jQuery if you want to use it. In most apps, its usually a must!

<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>

Step-4: Build the app.

Now, in your app.js, first add global variables that you’ll need throughout your app:

var GoogleAuth;
var SCOPE = 'https://www.googleapis.com/auth/drive'; //.metadata.readonly
SCOPE += " https://www.googleapis.com/auth/drive.install";
SCOPE += " https://www.googleapis.com/auth/drive.file";

var files = []; //array to store the list of files in user's drive.

Add the scopes as per your requirements, I had included auth/drive.install and auth/drive.file only because I had to provide an “open with this app” feature in the user’s own google drive interface so that they may visit our app and run their files through it, and thus use it as a “file opener”. If you don’t need to provide such extended features and just want full access to the user’s files, then the first one (auth/drive) is sufficient.

After that, write the entry point for our app in app.js. As per the script attribute, you’ll be redirected here by the Google authentication library:

/**
* This is the entry-point that gapi calls.
* 
* */
function handleClientLoad() {
 // Load the API's client and auth2 modules.
 // Call the initClient function after the modules load.
 gapi.load('client:auth2', initClient);
}

gapi.load() loads the auth library and prepares your app for authentication, then signals the control to initClient() function where you’ll perform the actual authentication like this:

 /**
 * starts the client authorization, most useful for debugging.
 * 
 * */
function initClient() {
    // Retrieve the discovery document for version 3 of Google Drive API.
    // In practice, your app can retrieve one or more discovery documents.
    var discoveryUrl = 'https://www.googleapis.com/discovery/v1/apis/drive/v3/rest';

    // Initialize the gapi.client object, which app uses to make API requests.
    // Get API key and client ID from API Console.
    // 'scope' field specifies space-delimited list of access scopes.
    gapi.client.init({
        apiKey: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
        discoveryDocs: [discoveryUrl],
        clientId: 'yyyyyyyyyy-yyyyyyyyyyyyyy.apps.googleusercontent.com',
        scope: SCOPE
    }).then(function () {
      console.log("init happened successfully.");
      GoogleAuth = gapi.auth2.getAuthInstance();
      
      console.log("now binding updateSigninStatus");
      GoogleAuth.isSignedIn.listen(updateSigninStatus); // Listen for sign-in state changes.

      // Listen for sign-in state changes.
      console.log("now calling updateSigninStatus");
      updateSigninStatus(GoogleAuth.isSignedIn.get());

      // Handle initial sign-in state. (Determine if user is already signed in.)
      //var user = GoogleAuth.currentUser.get();
      
      // Call handleAuthClick function when user clicks on
      //      "Sign In/Authorize" button.
      $('#sign-in-or-out-button').click(function() {
        handleAuthClick();
      }); 
      $('#revoke-access-button').click(function() {
        revokeAccess();
      }); 

    }, function(error){
        console.log("ERROR in gapi.init:", error);
    });
}

(Remember to replace the long string of “xxxxxx…” with your own API Key, and the long string of “yyyyyy….” with your own Client ID)

gapi.client.init() is an important function which does all the hard work of checking whether a user is authenticated, and if not, then redirect them to Google’s servers, perform the authentication, and if successful, redirect them back to your server and inside the .then() block in which you’ll handle your applications logic (like displaying the logged-in user’s name which you get using GoogleAuth.currentUser.get(), update the logged-in status (toggle sign-in and sign-out buttons accordingly), etc:

gapi.client is your global variable that could be used for doing all kinds of stuff. For example, this is how you can list the files available in the Google Drive’s root folder of the user as links in your document:

   gapi.client.drive.files.get({
        fileId: id,
        fields: 'name,webContentLink'
    }).then(function(success) {
        var webContentLink = success.result.webContentLink;
        var fileName = success.result.name;
        console.log("SUCCESS! name is: ", webContentLink);
        console.log("SUCCESS! webContentLink is: ", webContentLink);
        $source = $("<a href='" + webContentLink + "' >" + fileName + "<a>");
        $('body').append($source);
    }, function(fail){
        console.log(fail);
        console.log('Error '+ fail.result.error.message);
    });

The name field represents the file name and webContentLink is the actual link to the file in the user’s drive, you can use it however in your app once you get it (like download it using the AJAX jquery.get() method, etc.).

WordPress DIY: Adding twitter cards meta to your blog without using an external plugin

2018-07-08T00:00:00+00:00

Just like my last article, we won’t be focusing on using a third party plugin, but write our own plugin. I’m a minimalist and don’t prefer to use layer-2 solutions for really trivial things that can easily be achieved by writing code.

Now, though trivial to implement, the twitter cards support is a very important and useful thing for your blog. To understand why, consider the following example tweet from Python Software Foundation:

Links had no Twitter Card Meta

As you can see, the posted links on this tweet from the domain djangoproject.com didn’t expand into a preview because they didn’t have twitter card meta tags in their pages. Unlike Facebook who expands all posted links, Twitter doesn’t do that automatically, but only after parsing some meta tags which should be in their required format. Only after that, the tweets expand into a full preview like this:

Link had Twitter Card Meta

So as the owner of a WordPress blog, you’d be certainly interested in having the tweets containing links to your own site expand into these previews, right? So, let’s go about doing it.

Essentially, your web page should contain the following meta tags that twitter parses in order to come up with a preview:

<meta name="twitter:card" content="summary" />
 <meta name="twitter:site" content="@prahladyeri" />
 <meta name="twitter:creator" value="@prahladyeri" />
 <meta name="twitter:url" content="https://prahladyeri.github.io/blog/2018/06/people-migrating-from-github-to-gitlab-should-learn-about-these-details-first.html" />
 <meta name="twitter:title" content="People migrating from Github to Gitlab should learn about these details first" />
 <meta name="twitter:description" content="After &lt;a href=&quot;https://prahladyeri.github.io/blog/2018/06/microsofts-github-acquisition-an-unbiased-perspective.html&quot;&gt;Microsoft&#039;s recent acquisition of Github&lt;/a&gt;, a mass exodus has kind of begun and many small and large projects are moving their code bases to the much hyped &lt;a href=&quot;https://gitlab.com/&quot;&gt;Gitlab&lt;/a&gt; in a hurry, and these include both open and closed source projects. However, before migrating to Gitlab, they should take a pause and learn something about Gitlab and consider evaluating other alternatives too." />
 <meta name="twitter:image" content="https://secure.gravatar.com/avatar/3f253507b82dd33f352c08f649caaa54?rating=PG&size=75" />

All these meta tags have different meanings. For example, twitter:url is for the canonical URL of your page, twitter:title is for the title that should be displayed in the preview, etc.

Now, let’s add a simple PHP plugin file that automatically adds these tags in all the pages of our WordPress blog. Firstly, create a text file called “/wp-content/plugins/custom_headers.php” in your WordPress installation folder (no need to create if you had done already by following the last article).

After that, just add the below function and call it using add_action():

function add_twitter_card_header() {
    #twitter cards hack
    global $post;
    if(is_single() || is_page()) {
        $twitter_url    = get_permalink();
        $twitter_title  = get_the_title();
        $content = get_extended( $post->post_content );
        $attch_id = get_post_thumbnail_id($post->ID);
        $twitter_thumbs = wp_get_attachment_image_src($attch_id, full);
        $twitter_thumb  = $twitter_thumbs[0];
        if(!$twitter_thumb) {
            $twitter_thumb = 'https://secure.gravatar.com/avatar/3f253507b82dd33f352c08f649caaa54?rating=PG&size=75';
        }
        ?>
        <meta name="twitter:card" content="summary" />
        <meta name="twitter:site" content="@prahladyeri" />
        <meta name="twitter:creator" value="@prahladyeri" />
        <meta name="twitter:url" content="<?php echo $twitter_url; ?>" />
        <meta name="twitter:title" content="<?php echo $twitter_title; ?>" />
        <meta name="twitter:description" content="<?php echo esc_html($content['main']); ?>" />
        <meta name="twitter:image" content="<?php echo $twitter_thumb; ?>" />
        <?php
    }
}

add_action('wp_head', 'add_twitter_card_header');

This simple block of code will pick up all required things such as the post’s title content excerpt, etc. and supply them to twitter via the meta tags. Few important things you need to remember:

1. Firstly, update the $twitter_thumb variable in the following block with your own gravatar:

if(!$twitter_thumb) { $twitter_thumb = 'https://secure.gravatar.com/avatar/3f253507b82dd33f352c08f649caaa54?rating=PG&size=75'; }

This will show the gravatar by default if you haven’t set a featured image in your post.

2. You’ll need to add the “Read More” meta tag in all your posts, otherwise $content[‘main’] will return the whole thing instead of just the excerpt.

Once you do this and publish your post, head over to the Twitter Cards Validator Service and test your link. Its as easy as that!

Wordpress DIY: Adding Google Analytics Script to your Blog Without using an External Plugin

2018-07-07T00:00:00+00:00

Adding a custom script element to your Wordpress blog is really straightforward if you know what you are doing and there isn’t any need to install a third-party plugin for this.

Now, for something like spam protection (Akismet) or adding contact forms (Contact Form Seven), its quite understandable, but if you take the approach of adding third party plugins for every little plus and minus operation (such as adding a Google Analytics script), then its neither good for the maintenance nor security of your Wordpress blog.

In this article, I’m going to explain how to create a simple custom plugin for your Wordpress blog that you’ll hand-code yourself. The custom plugin will be quite generic and can be later used for adding other elements too such as maybe twitter card attributes, etc.

Create the following PHP Plugin file in the code editor of your choice and save it as /wp-content/plugins/custom_headers.php in your Wordpress installation.

<?php 
/**
 * @package Custom Headers
 * @version 0.1.0
 */
/*
Plugin Name: Custom Headers
Plugin URI: http://github.com/prahladyeri/custom-headers/
Description: Add a bunch of custom headers before the head tag.
Author: Prahlad Yeri
Version: 0.1.0
Author URI: https://prahladyeri.github.io
*/

function add_analytics_header() {
$analytics=<<<EOD
<-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-XXXXXXX-X">
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'UA-XXXXXXX-X');
</script>
EOD;
    echo $analytics;
}
add_action('wp_head', 'add_analytics_header');

Just remember to update the package meta stuff (Plugin Name, Author, etc.) as it suits you and put your own Analytics ID in place of UA-XXXXXXX-X.

Its as simple as that, your plugin is ready! Just enable this plugin by going to Plugins->Installed Plugins in your Wordpress admin panel.

Building Robust Single-Page Applications: Key Architectural Insights

2018-07-06T00:00:00+00:00

Lets face it, Web Apps are a complex and complicated beast, both on the front end and back end. The reason we had to come up with so many frameworks and libraries (angular/backbone/react/vue/marionette/require.js/etc.) is that the whole process is quite difficult and convoluted.

Once your app starts to scale in complexity, even the best of ideas and best practices in this field cannot prevent your app code from becoming a spaghetti of JavaScript and jQuery callbacks and DOM manipulation functions. Turning into a mess is essentially the nature of JavaScript when left to its own mechanism (event callback model and high asynchrony do ensure that). If you only use jQuery to architect your app, this is what it’ll soon end up becoming:

The Spaghetti Way

Once your app.js ends up with millions of these jQuery functions, it will soon become beyond understanding and sanity even for you as the author of the code. And that’s exactly why we need to understand this complexity and architect some structure and foundation in our apps.

One way of handling this complexity is the easy way - just delegate this whole thing to another complex beast of an opinionated framework such as angular or vue, and work with simple abstractions like angular views and controllers. This strategy does work to an extent (and you do get some mental sense of achievement too) but only as long as your app is limited in functionality and doesn’t scale in size and complexity. The problem with opinionated frameworks is that they are opinionated - they work only as long as your app is pigeonholed and fits in their own way of typically doing things, but the moment you step outside and want to do something that doesn’t confirm to that way (like rendering a complex DOM element or work with a difficult third-party UI library), then you are hit with a wall - unless you are prepared to venture too deeply into that framework and learn advanced stuff (like creating complex directives or providers in angular). But if you had to resort to that at the end, then why use an opinionated framework at all!

The Opinionated Framework Way

As you can see in above diagram, using an opinionated framework is like driving a car with automatic gear system. You cannot control the speed or acceleration, nor can you synchronize the clutch action, its the black box of the automatic gear that does it for you. But unlike this simple gearbox analogy where speed and clutch synchrony are the only two variables, your web app has lots of variables, so the chances of going wrong and getting stuck with this approach drastically increases as your web app starts to scale in complexity. Manually doing things using the Backbone way may appear to be difficult or complex initially, but that’s a much better way than trading off that complexity for an opinionated black-box framework about who’s inner workings you don’t understand anything at all.

The first step towards building a single page app should be deciding what your app is going to do and how its going to do it. Every app is different in features, functionality and work flow, hence it pays to use an uber-light framework such as Backbone instead of any heavy and opinionated ones. Backbone in the frameworks world is pretty much like what jQuery is in the libraries world, it doesn’t do much on its own besides giving you methods and objects for creating a layer of structure and organization (just as jQuery doesn’t do much on its own besides adding a sugar-syntax wrapper for native DOM manipulation functions of JavaScript).

Now, because of this exact flexibility and freedom, there is no one correct way to build Backbone apps. Backbone is heavy in philosophy and light in implementation, and the philosophy being simple:

Separation of concerns (models, views and other components should be kept separate).
Separation of roles (Organizers and implementors should do their own thing, one shouldn’t step into the shoes of others).

Personally, I think that if you follow only these two principles sincerely and stick to them, then you can handle whatever amount complexity life throws at your app. Of course, to architect such an app is an art in itself and you may not get it right the first time. The architecture itself may even need to evolve with time and complexity as your app grows and scales. One global event coordinator (or organizer) can be enough initially, but later on, you may have to add a DOM coordinator for handling DOM events, a data coordinator for handling connections to the database objects, etc. as you scale.

Here is a basic example architecture that you can probably use for an app of low to medium complexity:

The Backbone Way

And this is an example implementation which I’m developing as a side project. Of course, the app doesn’t do much presently besides user management and routing to dummy pages,

Of course, this isn’t the only Backbone way to build apps, but I think its a good one for a start. The primary problem you’ll face if you stick to doing things your own way and not use an opinionated framework is that of organization. You need an organizer/coordinator object to co-ordinate with various components of your app and keep them in sync with each other and this is where the Backbone.Events API provided by the framework shines. The Backbone.View object also plays a great part in ensuring the separation of DOM manipulation code from rest of your app, and along with the underscore template library, it becomes a power combination for rendering DOM! With underscore templates, you can actually use javascript functions and variables inside a template, not just clumsy tags like ng-something!

<% if(modal['buttons']) { %>
  <div class="ui-button-container">
	<% _(model['buttons']).each(function(button) { %>
	  <a class="ui-button ui-button-pill <%= button.extra_class %> " href="<%= button.href %>">
		<span class="label">
		  <span class="section"><%= button.label %></span> 
		</span>
	  </a>
	<% }) %>
  </div>
<% } %>

Of course, its not a very good practice to mix javascript with templates, but it just shows the power and flexibility of using backbone+underscore compared to other vendor frameworks.

Ultimately, it all comes down to your preference, you can use angular or vue or react if you really want to. However, always understand the reason why you are hooking to a third party framework. If it is just for escaping the complexity of your app, then no amount of framework or libraries in the world are going to help you. Passing on the complexity to a black box like angular or vue is just a band-aid solution that will fall apart the moment you scale in complexity and you’ll get stuck by the limitations of the framework. At that point, you’ll have to make one of these two decisions:

1. Abandon the framework and do everything right from scratch using the manual way of Backbone.

2. Understand the framework internals too deeply and customize it to achieve what you want (but then what was the point of using this framework in the first place?)

Ultimately, its your decision to choose a framework. Consider it wisely after weighing in all the pros and cons. Best of luck!

What problem is React/JSX solving in your App?

2018-07-05T00:00:00+00:00

Framework wars and debates are very much a thing these days, be it Angular vs Backbone or Angular vs React, but the real debate isn’t about these frameworks. The real debate essentially comes down to which is the most efficient way of structuring your app and more importantly, rendering and managing your DOM.

Whilst the traditional jQuery/Backbone way is to render the DOM directly using methods like $(component).html(“DOM Code”), React considers it an anti-pattern and recommends the way of virtual DOM - a concept where you don’t render the DOM directly as it is, but keep a virtual copy of it and render only the differential which is essentially patching the DOM.

One thing that comes to mind is that more than an improvement over the former approach, you are basically trading off CPU overhead for lesser memory usage. The former jQuery approach is more memory intensive as the browser needs to keep large amount of DOM structural code in memory until its rendered (yeah, it gets pretty large in a non-trivial app with lots of widgets that may need to be rendered in a complex array of patterns). However, what exactly are we gaining by using the JSX virtual DOM method? The JSX approach is just as much CPU intensive as the jQuery approach is RAM intensive because it takes just as many CPU cycles to compute the differential and come up with a “patch” version of the DOM. In fact, the trade-off may be even worse considering that React is more of a library overhead than jQuery which is just a light “write less do more” wrapper over JavaScript DOM manipulation functions.

The real question we should be asking ourselves is why do we keep looking for that holy-grail JavaScript framework every now and then? Yesterday it was angularjs, today its React and tomorrow it will be vue.js. Instead of running after the shiny new framework, why not sit down and see what you are doing wrong with the present tools you have. After all, these tools are industry proven and they still exist for a reason.

I think most of the issues with using jQuery come from incorrect usage patterns than any problems associated with actual rendering. Consider the following often used pattern for rendering view blocks:

$.get("partials/navbarTemplate.html", function(data){
  $("#div-navbar").html(data); //render a template
  ..
});

This coding pattern is typically abhorred by most React experts as they feel that rendering a whole bunch of html code (data argument in this case) is an anti pattern as the browser has to repeatedly render a lot of boiler-plate DOM unnecessarily. However, that’s only true if you do this often times and repeatedly call this function. If you use a good architectural pattern for structuring your app (like the one provided by Backbone.js), you can render this same DOM in a very idempotent way:

app.NavbarView = Backbone.View.extend({
    el: "#div-navbar",
    initialize: function() {
        var temp = this;
        $.get("partials/navbarTemplate.html", function(e){
            temp.template = _.template(e, {});
        });
    },
    render: function() {
        this.$el.html(this.template());
    },

});

In above code, we are still using jQuery.get, but in a more organized and structured way. Firstly, we call jQuery.get only in the initialization of the view to get the template and store it in the cache, so the network overhead isn’t involved each time we need to use a template. Furthermore, backbone itself caches and stores the block element to be rendered (this.$el), so that we don’t have to trouble the browser with other areas of the DOM where rendering isn’t required. Finally, we just call the underscore template and render the element:

this.$el.html(this.template());

Of course, this could be probably further optimized by using jQuery.empty().append() instead of jQuery.html() if you want to tweak the last drop of performance!

this.$el.empty().html(this.template());

However, I don’t think this kind of premature optimization is really needed unless you are building a really complex app and even then this isn’t required in about 90% of the cases, the browsers have become considerably fast in recent years, at least in the area of DOM rendering.

And this is exactly why I think that using virtual DOM libraries like React is an anti-pattern. You are essentially stepping into the shoes of the browser, isn’t it? If the kind of partial patching implemented by React is really efficient, wouldn’t the browsers be doing it themselves? Maybe they will take some of the best ideas from React and JSX, and implement it themselves in the coming future, but why should you (as a programmer) be bothered with that is what I don’t understand.

This whole debate about frameworks is pretty much centered on separation of concerns essentially (models, views and controllers/organizers should all be cleanly separated) and to some extent, React is going against that separation by doing what the browser is supposed to do.

Notes on open source vs free software

2018-07-03T00:00:00+00:00

Socialism and Communism have a lot in common, both ideologies aim to bring an equilibrium and welfare in the society by removing the income inequality between the rich and the poor. However, how they both go about doing that job is what makes the difference between sky and earth. Whilst Communism does advocate the use of force to achieve its aim, Socialism does not and that’s the major difference of spirit between them. If you remove this force element, you’ll find that they are both pretty much the same thing with only technical differences here and there.

Coming to the world of software, it’s a similar case when it comes to GPL (free software or copyleft) and non-GPL (open source or liberal) variety of FOSS licenses. The GNU GPL forces their ideology upon the user of the software by expressly restricting him/her to the kind of use cases permitted on the said software:

> To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.

This restriction or force factor is what makes GPL or LGPL less desirable than the other more liberal (also called “open source”) licenses, just as the communist ideology is less desirable than socialist ideology by most people.

On the other hand, consider a liberal open source license such as the MIT license. This license doesn’t force anything on how the software may be used by the user (except that the license header itself carrying the name of the author should be passed around with any distributed copies):

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

They may use it in any kind of use cases they please, even include it in a closed source or proprietary software if they want. Similar is the case with other open source licenses such as the BSD Modified licenseor the Apache 2 license. None of these licenses force their users into any kind of restrictions, which is probably why they are more popular and used than the copyleft variety of licenses today.

Finally, coming to the point of contrast between the copyleft and open source licenses, you’ll find that they are all ultimately trying to achieve more or less the same goal, though some would strive for an excellence methodology in software development while others would term it as a struggle for users’ freedom depending on what viewpoint you choose to look at it.

Now, the reasons for GPL restrictions and their ideology are quite understandable, but the way they go about implementing it is certainly not ideal from a practical standpoint. Stallman’s approach of making copyleft software unusable when linked with non-GPL or proprietary code is like a child’s tit-for-tat attitude. Just as proprietary software companies won’t let you use their software without a license and make it unapproachable to those who don’t have it, Stallman wants to make the GPL world unapproachable to the proprietary companies by making this restriction on mixing GPL with proprietary code. In a perfect world (consisting of mostly power users who would rather visit support forums and do their own builds than use proprietary software), this may have worked wonders to Stallman’s cause, but unfortunately we don’t live in such a perfect world!

On the other hand, the open source or liberal licenses that don’t put any restrictions on its usage serve the purpose of everyone and so everyone is happy and thrives in this world. Power users are happy because they no longer have to use privacy invasive and/or bloated proprietary software as they can compile their own builds or even visit support forums or issue trackers of various open source projects if required. On the other hand, proprietary software companies are also happy as they get to use the fruits (source code) of open source contributors without spending anything on their part. Of course, taking the moral cognizance of these fruits, many of them also do tend to contribute back by either financially supporting other open source projects or releasing their in-house developed source code as open source to the world at large. There are several examples of these such as Microsoft’s Visual Studio Code, Adobe’s code editor called Brackets.io, Apple’s Swift, etc. Now, consider that in a Richard Stallman’s “pureland GPL” world, we wouldn’t have all these contributions back from the proprietary companies!

Webpack Project has a Vulnerability

2018-06-28T00:00:00+00:00

The other day, I was going through this medium post which describes the kind of chaos and insecurity currently plaguing the JavaScript world, and the numero uno reason for that is the astronomical number of npm packages.

When you usually install a non-trivial library or application through a package manager, the expectation is that the number of dependencies should be as less as possible. This not only helps you in managing the disk space (very important for cloud hosting), but also makes a manageable code audit possible. For instance, the Python’s Flask package (which is a considerably large web framework, a lot more complex than Webpack which is just a JavaScript bundler) has the following four dependencies:

 install_requires=[
 'Werkzeug>=0.14',
 'Jinja2>=2.10',
 'itsdangerous>=0.24',
 'click>=5.1',
],

On the other hand, this trivial Webpack package on NPM has an astounding 25 dependencies on various 3rd party packages:

@webassemblyjs/ast
 @webassemblyjs/helper-module-context
 @webassemblyjs/wasm-edit
 @webassemblyjs/wasm-opt
 @webassemblyjs/wasm-parser
 acorn
 acorn-dynamic-import
 ajv
 ajv-keywords
 chrome-trace-event
 enhanced-resolve
 eslint-scope
 json-parse-better-errors
 loader-runner
 loader-utils
 memory-fs
 micromatch
 mkdirp
 neo-async
 node-libs-browser
 schema-utils
 tapable
 uglifyjs-webpack-plugin
 watchpack
 webpack-sources

However, your worry hasn’t even started yet, your real worry starts when you realize that the other super-trivial package mentioned in that article called is-odd has a whopping statistics of 1.4 million downloads per week:

{.alignnone .size-full .wp-image-699 width=”453” height=”445”}

Now, any programmer worth his salt will know that its a one line coding effort to determine whether a given number is odd or even, even in JavaScript:

function isEven(n) {
 return n % 2 == 0;
}

function isOdd(n) {
 return Math.abs(n % 2) == 1;
}

Now, our master programmer who developed this “is-odd as a service” not only goes ahead and registers whole new npm packages called is-odd and is-even, but also makes a lot of his other packages depend on it. Can you even begin to imagine what kind of bureaucratic politician you have to be in order to do that!

Now, this in itself couldn’t have caused any problem, the real issue here is that the highly reputed and popular Webpack project is also using one of his packages (for doing a trivial regular expression check on a string) and the dependency chain is such that the following four packages are also pulled in whenever you install Webpack from npm since Webpack depends on them (and this is what explains is-odd’s 1.4 million weekly download figure):

is-odd -> nanomatch -> micromatch -> webpack

Now, even a CS undergraduate should be able to see the management nightmare, not only of burdening your hard disk space of these additional packages whenever you npm install webpack, but also the security nightmares associated with it. In future, if the developer of is-odd package clubbed some malware in his code and propagated it throughout the node-ecosystem, can you even begin to imagine the disaster its going to cause on your production machines? I agree that this applies to other packaging systems like pip and composer too, but problem here is that the number of npm packages is too large and too unmanageable.

And to top it, the developers in the ecosystem are paying no heed to this, they think this is something to be proud of and worthy of chest thumping. They should try to understand that DRY (Don’t Repeat Yourself) becomes a self-harming pattern beyond a certain extent, especially when you start releasing packages like is-odd and is-even. They get argumentative and counter you with a militant defense when you try to explain this point to them. For instance, I raised an issue on Webpack’s Github repository a few days ago for this exact problem (remove dependency from micromatch package) and they simply closed it down giving some hilarious arguments. Their arguments are really interesting, but beyond sanity for someone who values security and maintainability of production applications.

Smyte is no more - The latest episode in the acquisition saga of Tech Giants

2018-06-22T00:00:00+00:00

Not even a month has been passed since Microsoft gave a big surprise to the world at large and the open source community by acquiring Github Inc. a few days ago, and there has been another acquisition yesterday. This time, Twitter Inc. bought the well known online service Smyte. Smyte was an online provider of anti-abuse technology services, i.e. things like AI algorithms for identifying online trolling, etc. Many customers were totally dependent on this provider’s API which suddenly disappeared off the radar after Twitter Inc. acquired them yesterday.

A whole lot of customers and other depending services were affected because of this including npm, the popular Node.js package manager. As usual, Reddit was one of the first places where people turned up to know what’s going on and update their statuses. As one of the comments quite intelligently notes, this is a very good lesson on this modern web development way that involves depending on a zillion external services and leaving your application open to the whims of acquisitions and centralization of technology powers:

::: {.reddit-embed data-embed-media=”www.redditmedia.com” data-embed-parent=”false” data-embed-live=”true” data-embed-uuid=”5bebab31-c06c-4d2c-bbd0-d938f9f97409” data-embed-created=”2018-06-22T14:47:54.034Z”} Comment from discussion Reminder that external dependencies can shut down without notice. :::

Similarly, Twitter was also filled today with tweets of complains from existing Smyte customers who have suddenly stopped receiving any communication from their service provider:

https://twitter.com/arthens/status/1009925187364458496

Jeremy Ashkenas, the inventor of famous Backbone.js and Underscore.js JavaScript libraries expressed his feelings in a very sarcastic tweet:

https://twitter.com/jashkenas/status/1010014139320680449

I don’t know what was Twitter’s intention before acquiring Smyte, but the way this is going on, this has to end in only one of the two ways:

1. Either developers take lessons from these incidents and stop depending so much on layer-2 solutions and external dependencies.

2. Technology giants become a bit more empathetic to the needs of smaller businesses and understand that just acquiring other services isn’t a solution to the innovation problem.

People migrating from Github to Gitlab should learn about these details first

2018-06-11T00:00:00+00:00

After Microsoft’s recent acquisition of Github, a mass exodus has kind of begun and many small and large projects are moving their code bases to the much hyped Gitlab in a hurry, and these include both open and closed source projects. However, before migrating to Gitlab, they should take a pause and learn something about Gitlab and consider evaluating other alternatives too.

Gitlab Stack

According to above StackShare.io chart, Gitlab basically runs on Microsoft Azure cloud hosting facility. So, if you are leaving Github in order to escape the clutches of Microsoft, then you are headed to a totally wrong place! Microsoft is exactly the place where your source code will reside in this case, only difference is that instead of controlling it directly (as in the case of Github), Microsoft will be controlling your code only technically.

I know, some of you will be saying that you are self-hosting an open source copy of Gitlab and not actually moving to Gitlab.com. In that case, please have a look at another item in that stack, namely Rails (RoR or Ruby on Rails framework).

Though Rails is a great framework that developers enjoy to code with, its a performance hog when it comes to actually running on production! There is a reason why Twitter ditched Rails in favor of Node.js instead of fixing the interpreter like Facebook did with PHP. Apart from Rails being a performance hog, consider that a git hosting facility is not a simple CRUD app. Its very difficult to do advanced things like CI/CD right in a framework like Rails and the effect is showing. It may work out initially, but once your code base starts to increase and your integrations start to scale, you’ll hit the RoR scaling limit sooner or later, like many others have. Its not uncommon for Gitlab to eat gigabytes of your RAM or consume 100% CPU. So, if you are trying to host Gitlab in a small Digital Ocean droplet or Amazon AWS Micro instance, you can just forget about it!

Or, you can sit back and evaluate your options, it really depends on what you basically want. If you just want a free git hosting facility and don’t want to self-host, there is already Github. If you don’t like Microsoft, then you have Bitbucket, SourceForge, Debian Salsa and others too apart from Gitlab, so consider those options too before blindly deciding on Gitlab and falling for their marketing trap.

On the other hand, if you are ready to self-host and have a smaller budget (just for an AWS Micro or Digital Ocean droplet, for instance), then you can use one of the several open source and light-weight git hosting software like gogs, gitea, phabricator and many others and self-host.

Finally, if you have a budget for hosting Gitlab on a larger instance (like AWS Large instance or 2GB droplet from Digital Ocean), then the first question I’d ask you is why not just stick to paid hosting plans of Github (or Gitlab/Bitbucket if you don’t like Microsoft). That will be a lot cheaper and lenient on your pockets than self hosting a copy of Gitlab on a larger instance.

Whatever route you end up choosing, it should be a calmly taken logical decision after due consideration to all facts, not in this Github acquisition frenzy. All the best.

Links:

Microsoft's Github acquisition - A perspective

2018-06-08T00:00:00+00:00

As someone who has worked on Microsoft tooling since the days of Foxpro 2.6 to Visual Studio 2010 in IT career, and yet ditched them all for PHP, Python and open source years later, I think I am qualified to offer a somewhat neutral or objective perspective on this acquisition.

When this news was first announced on last Friday, I was naturally puzzled and so were a lot of other developers and not without reasons. The way this was initially announced without any clarification about what they were going to do with the hosting facility in future, speculations were bound to be raised and people were bound to be pissed off, especially given Microsoft’s history of being “not so friendly” with open source and “not so good” with some acquisitions.

When it comes to open source, the Linux subreddit is the place where developers pour their hearts out and this particular thread quite summed up their initial knee-jerk reaction. As the most upvoted comment says:

its like Microsoft is obsessed with generating as much frustration in the world as possible

Other reactions were also quite similar:

I LOL’ed.

First, Sourceforge started to install malware on the open source projects. So everyone run to Github. Now, Github might get owned by Microsoft. Not to mention Github is 100% closed source. Gitlab looks good every day. There are other open source git solution too.

GitLab, the top open source competitor to Github soon became the most discussed alternative (apparently, their marketing team did their bit too with the perfect timing!). And so, a mass exodus of projects soon began, which not only became the most discussed topic on reddit, but also caught a lot of media attention too.

Just within a couple of days from this announcement, some 13,000 projects had already migrated to Gitlab which included some prominent names. Naturally, it was time now for Microsoft to do some damage control. But if their PR team was any wiser, they should have already anticipated this and this damage control move should have been done at the outset as the first thing. How could they not expect this backlash considering Microsoft’s past history with open source (however distant it may be)?

The first move came when Satya Nadella, the Microsoft CEO gave an interview to CNBC on Monday (4th June):

One of the most important things Satya said in the interview was that Microsoft is a developer tooling** **company at core (let’s ignore Windows and Office for a moment!). However, most developers still had concerns, they wanted to know how this high level strategy of being open source developer friendly will turn out to be in practice.

Most importantly, they needed an assurance that their daily driver for source control isn’t going to be integrated and hijacked by other Microsoft products like Skype, Linkedin or even a Passport account. And that assurance came yesterday from Nat Friedman, the future CEO of Github in his Reddit AMA.

Firstly, the fact that Github Inc. was going in the hands of an open source veteran who had contributed to GNOME and MONO projects in the past was itself quite reassuring. At least, Github isn’t going to be controlled by a typical corporate honcho who has nothing but shareholders’ interests in mind! Nat assured most redditors that Github is going to stay as it is, its developer focus is not going to be shifted, nor is it going to be integrated with any other products. And most importantly, they will never require a Microsoft account to login to Github, rather, their other products might consider allowing a Github user to authenticate to their systems in future:

We love GitHub login. Your GitHub account is your developer identity, and many users are accustomed to signing into developer tools and services (e.g. Travis, Circle) with their GitHub accounts. So, if anything, we may decide to add GitHub as a login option to Microsoft.

Nat also goes on to further assure redditors in the AMA that they will always remain a “developer first” company, and are keen to learn a lot from Github from this acquisition. Rather than impose their own work culture on Github, they’ll be taking lessons from Github and try and be like them:

We bought GitHub because we appreciate how special it is. That’s why we have two principles for this acquisition going forward:

Developers first. We will evaluate every decision through the lens of what is best for developers. This includes GitHub’s status as an open platform with open APIs that any developer can use to extend GitHub’s functionality. And it includes our commitment that we will support developers on GitHub in their use of any language, any license, any operating system, any device, and any cloud.

Independence. We are not buying GitHub to turn it into Microsoft; we are buying GitHub because we believe in the importance of developers, and in GitHub’s unique role in the developer community. Our goal is to help GitHub be better at being GitHub, and if anything, to help Microsoft be a little more like GitHub.

All in all, this damage control or whatever Microsoft has done seems to have done the magic at least for the moment. People and projects seem to have stopped their exodus to Gitlab, though its difficult to say what could happen in the long term.

How to use Bash+rsync to automate your periodical backups on Linux

2018-03-04T00:00:00+00:00

Linux is all about simplicity. Simple and time-tested tools like iptables, netstat and rsync can be called for help for basic tasks instead of untrusted third-party tools as happens in case of Windows. If you are a desktop user, then taking frequent backups of your data to a pen-drive or external disk is a typical problem to solve.

In this article, I’ll show how I solved this problem using a combination of bash scripting and rsync, the basic tooling available on any linux distro these days, be it Ubuntu, Debian or Fedora.

One of the things you may want to do is determine what folder(s) you want to backup to which device. You may want to copy the source code folder only to your pen drive and your images and documents to only external drive, for instance. Here is where identifying the disk label through bash comes in handy in the backup script:

if [ -d "/media/prahlad/DATA128" ]; then
	device_name="DATA128"
	folder_name="/media/prahlad/DATA128"
elif [ -d "/media/prahlad/extHD" ]; then
	device_name="extHD"
	folder_name="/media/prahlad/extHD"
else
	echo "No Drive Found"
	exit
fi

In the above example, DATA128 is a pen-drive and extHD is an external hard-drive and the above script determines which disk is inserted in the USB drive. You can then use the $folder_name bash variable to dynamically copy to that device instead of hard-coding that path unnecessarily. You can also use $device_name bash variable to include or skip specific folders when running the rsync command:

rsync -va /home/prahlad/source "$folder_name/source"

if [ "$device_name" = "extHD" ]; then
 rsync -va /home/prahlad/Pictures "$folder_name/home/Pictures"
 rsync -va /home/prahlad/Documents "$folder_name/home/Documents"
fi

The rsync command itself is also versatile enough to do a lot of things which are not possible using a simple copy-paste-replace using a file manager. For instance, the “a” or “–archive” option intelligently archives (copies) files while skipping identical ones based on checksum or modification date automatically. Further, the “–delete” option deletes files which are present on the destination backup device, but not on the source device which is typically the case when you want to backup your data. Run “man rsync” to see the full range of options exposed by this wonderful command.

Finally, another advantage of using a script for backup automation is that you can implement custom actions through the script. For example, compressing the mozilla firefox user folder before taking its backup:

echo "backing up firefox..."
tar czf ~/firefox-backup.tar.gz ~/.mozilla/firefox/
rsync -va ~/firefox-backup.tar.gz "$folder_name/home/firefox-backup.tar.gz"

Below is a typical example of how you might implement a script to automate backup of your home folder and set this as a cron job to run say weekly or fortnightly:

#!/bin/sh

if [ -d "/media/prahlad/DATA128" ]; then
 device_name="DATA128"
 folder_name="/media/prahlad/DATA128"
elif [ -d "/media/prahlad/extHD" ]; then
 device_name="extHD"
 folder_name="/media/prahlad/extHD"
else
 echo "No Drive Found"
 exit
fi

echo "Device: $device_name"
echo "Folder: $folder_name"

#start copying 

#home
rsync -va ~/.bashrc "$folder_name/home/" --delete
rsync -va ~/.profile "$folder_name/home/" --delete
rsync -va ~/Downloads/ "$folder_name/home/Downloads/" --delete
rsync -va ~/Documents/ "$folder_name/home/Documents/" --delete
rsync -va ~/Pictures/ "$folder_name/home/Pictures/" --delete
rsync -va ~/.ssh/ "$folder_name/home/.ssh/" --delete
rsync -va ~/.gnupg/ "$folder_name/home/.gnupg/" --delete
rsync -va ~/.thunderbird/ "$folder_name/home/.thunderbird/" --delete
rsync -va ~/.mozilla/ "$folder_name/home/.mozilla/" --delete

echo "Done"

You can customize the above script by adding custom actions or if conditions to skip or include specific folders based on the device label.

Tux Drive - A Command Line Tool to Access Google Drive from Linux

2017-09-20T00:00:00+00:00

One of the most boring things we need to perform in life is using the Google Drive. On one hand, so useful is this tool provided by Google (so many cloud GBs for free, yay!) but on the other hand, the web interface to access those files is not quite as intuitive, especially so for power users, and certainly so for linux users!

Unlike the good old FTP/SFTP where you could just run some geeky commands in a terminal (or use a super-intuitive GUI like FileZilla to just drag and drop files), the Google Drive web interface results in a crippled work flow. To top it, we don’t even have a desktop client like the Windows folks do.

Tux Drive

To solve this problem, I have written a console based python program called tuxdrive.

{width=”687” height=”302”}

Tux Drive running on Ubuntu 16.04 LTS

Its nowhere as comprehensive as FTP/SFTP in flexibility (yet), but it does some basic things like ls (or dir) to list remote files and directories on your drive, get file_name to download a file, cd remote_dir to change the drive directory, etc.

Here is the entire list of vocabulary that tuxdrive is currently acquainted with:

 help (or ?): Shows this help facility.
 dir (or ls): Lists all files and folders on drive.
 !dir (or !ls): Lists all files and folders in current directory.
 get (or pull) <item>: Pulls the named file/folder from drive to current working directory.
 put (or push) <item>: Pushes the named file/folder from current working directory to drive.
 rm <item>: Delete the named file/folder on remote path.
 pwd: Print working directory (remote/drive).
 cd: Change working directory (remote/drive).
 lpwd: Print working directory (local).
 lcd: Change working directory (local).
 rdcache: Show remote directory mapping of id and folder paths.
 rfcache: Show remote files mapping of id and folder paths.
 mkdir: Create a directory on remote path.
 exit: Exits this program.

Instructions to download and run tuxdrive can be found on the project github. I sincerely hope this tool becomes useful to as many users as possible. If something good comes out of this, I’ll think about integrating dropbox too into this in future.

Have a Good Day.

How To Make Your Ubuntu Desktop Faster

2017-09-19T00:00:00+00:00

The usual advice you get when seeking a more resource conserving distro is to use a lighter spin-off such as Xubuntu, Lubuntu, etc. However, not many people know that even a “heavy” distro such as Ubuntu LTS with the default Unity Desktop could be made much lighter by uninstalling some packages and removing others from the startup. Let us see how it can be achieved:

Remove unwanted items from Startup Applications

When you go to the Ubuntu Dash and start the “Startup Applications” dialog, it doesn’t show you the whole picture. There are still many “hidden apps” that silently start in the background without you knowing it. In order for all these hidden apps to be displayed in that dialog, you’ll have to run the following command in the terminal once:

sudo sed -i 's/NoDisplay=true/NoDisplay=false/g' /etc/xdg/autostart/*.desktop

Once you run this and then open the “Startup Applications” dialog, you’ll be able to see the hidden apps too:

Startup Applications (Ubuntu)

From this dialog, you can disable an app from starting up by “unchecking” it. The most important that I typically uncheck on a new installation are Backup Monitor and Desktop Sharing. It doesn’t make sense to have such a huge backup schizophrenia on a non-production desktop computer, so I wonder why the Backup Monitor is enabled in the first place. About 99% of sysadmins either perform a manual backup periodically, or write an automation script use something like rsync which is specifically designed for the job.

Desktop Sharing is an equally unnecessary tool to have been enabled by default on a linux desktop. This ain’t the Windows world where people use Remote Desktop clients to connect to other PCs on the same network. About 99% of linux folks use ssh to connect to remote machines instead. Desktop Sharing should be an opt-in feature in any case and a user who specifically needs it can enable it when needed.

Remove unnecessary background services such as cups and avahi-daemon

On a fresh Ubuntu installation, run sudo netstat -antpe and you’ll see the rarely used cupsd daemon (common unix printing service) running on a free TCP port and silently leeching your memory and network resources. Similarly, you can run sudo netstat -anupe to scan for UDP ports and you’ll similarly see the cups browser daemon (cups-browsed) and the avahi daemon (avahi-daemon) services.

Use of printers has become less and less, especially since the oncoming of digital age and internet revolution. Heck, even companies have been encouraging email communication these days in a bid to avoid paper work and save trees, and here we are - one of the top linux distributions running a printer service by default!

My humble request to them is please make this an opt-in feature, those who need it will have enough sense to run sudo apt install cups. Until that happens, the rest of the folks can do the following:

sudo systemctl disable cupsd
sudo systemctl disable cups-browsed

We are not uninstalling it, just disabling the service just in case. And before you say that cups is needed for PDF exports from browsers, then no, the latest versions of firefox and chrome come with their own PDF extensions and don’t depend on this service anymore.

As for the other service (avahi-daemon), it really doesn’t make any sense as avahi is the linux implementation of Apple’s proprietary protocols for peer-to-peer communication between their iDevices. Nobody in their right minds would be using an open source OS like Ubuntu on their PC/Laptop to communicate with proprietary Apple products! The two just don’t play well together as Apple products only work in a closed eco-system.

As for those rare folks who absolutely need avahi-daemon, let them install it themselves, why include this by default? Until that happens, the rest of us can disable avahi-daemon by simply running:

sudo systemctl disable avahi-daemon

Disable HUD (if you don’t use it)

One of the reasons why Xubuntu has such a low memory footprint (~200 MB at idle) is that it doesn’t come with memory leeching services such as the HUD. For example, on my laptop, HUD service eats a good 30MB of RAM which is too much for a service that I don’t even use. As described in this post, you can disable the HUD service from the terminal as follows:

sudo chmod -x /usr/lib/indicator-appmenu/hud-service # 32bit systems OR
sudo chmod -x /usr/lib/x86_64-linux-gnu/hud/hud-service # 64bit systems

Disable evolution processes (if you use thunderbird or anything else instead)

Evolution processes(such as evolution-calendar-factory and evolution-addressbook-factory) are huge memory leechers and a drain on your RAM. If you just start the system monitor and search for evolution, you’ll find 4-5 different processes consuming a good 120MB of your RAM! Now, for someone who doesn’t even use the evolution email client or any of these services, why should they be there in the first place? But unfortunately, you cannot just remove (uninstall) these packages using apt. Trying to apt remove evolution-data-server will break your system as they’ve made it a core part of the desktop. What you can do instead (if you really want to claim back that RAM) is a workaround suggested in this post. Simply rename the folders as follows:

sudo mv /usr/lib/evolution-data-server /usr/lib/evolution-data-server-disabled
sudo mv /usr/lib/evolution /usr/lib/evolution-disabled

Optionally, a less dirty hack is to remove the executable flag from the individual processes, so that they won’t start. This way, the processes won’t start again if you update your core packages in future.

sudo chmod -x /usr/lib/evolution/evolution-calendar-factory # less dirty hack

Remove GNOME software center (only if you don’t use it)

This piece of junk (gnome software center) takes a good 70-80MB on any typical ubuntu installation. For managing software, I find the good old apt install way to be much more intuitive than the gnome software center. For those rare cases when you absolutely need a GUI, the synaptic package manager works absolutely fine and without taking a constant toll on your resources.

As described in this answer, all it takes to safely remove gnome software center from your system is:

sudo apt purge gnome-software

After trimming the programs and services as mentioned above, your system should become nearly as light-weight as those other lighter distros like xubuntu, lubuntu, etc. and you’ll still be able to enjoy and work with a richer and more visually pleasant user interface!

On my own laptop, I was able to reduce the idle memory consumption from ~550MB to ~300MB which is as close as it gets to ubuntu-mate!

Update

As of 18.04 LTS, Ubuntu has replaced gnome software center with a new piece of crap called “snappy” which is no better than its predecessor! To remove snappy, just follow this process:

#check what snaps are installed
snap list

#remove all snaps
sudo apt purge snapd

#delete the snap folder
rm -rf snap 

#optionally replace snaps with standard apps
sudo apt install gnome-calculator gnome-logs gnome-characters gnome-system-monitor 

I still don’t understand why ubuntu keeps shipping this extra needless junk like cups and avahi with each new release (considering its useless for the average or most typical user out there). My best guess is they do this so that the user googles for ways to remove it and thus improve their linux skills!

Reference:

Effortlessly import Disqus comments into WordPress: a step-by-step guide

2017-09-07T00:00:00+00:00

Why Migrate from Disqus to WordPress?

Some time ago, I had switched from a self-hosted WordPress blog to a statically generated (Jekyll) blog hosted on Github Pages. For a commenting system, Disqus was quite an easy choice at that time since it was zero hassle for us site owners, and Disqus did all the heavy lifting from filtering the comments to storing and displaying them.

But as time went on, I started realizing that implementing a static blog was not quite the right thing. Firstly, there were privacy issues around Disqus because of which many readers of my blog were discouraged from commenting. Secondly, the concept of a “static site” itself felt quite constraining to me as I couldn’t implement things like contact form or a questionnaire to interact with my viewers. As a result, I decided to switch back to a plain old self-hosted WordPress blog.

Migrating Jekyll posts to WordPress

Now, importing the posts was quite straightforward using the Jekyll generated RSS feed link that was pretty straightforward to use. In case you don’t have it in your Jekyll blog, it’s very easy to write one using liquid template. Just create a file named rss.xml in your root folder with below contents:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
 <channel>
 <title>{{ site.name | xml_escape }} - Articles</title>
 <description>{% if site.description %}{{ site.description | xml_escape }}{% endif %}</description>
 <link>
 {{ site.url }}</link>
 {% for post in site.posts %}
 {% unless post.link %}
 <item>
 <title>{{ post.title | xml_escape }}</title>
 <description>{{ post.content | xml_escape }}</description>
 <pubDate>{{ post.date | date: "%a, %d %b %Y %H:%M:%S %z" }}</pubDate>
 <link>
 {{ site.url }}{{ post.url }}</link>
 <guid isPermaLink="true">{{ site.url }}{{ post.url }}</guid>
 </item>
 {% endunless %}
 {% endfor %}
 </channel>
</rss>

Once you generate the blog, you can import all posts into your WordPress by referring to the /rss.xml url on your existing Jekyll blog.

Importing Disqus Comments

However, the bigger issue here was importing the Disqus comments, becuase while Disqus does allow you to export a dump of your site comments, their XML format is pretty weird and isn’t the standard one used by WordPress and other blogging systems, as a result of which there aren’t too many ready tools for importing comments from this format to any other system.

As a result, I had to write my own WordPress importer tool. Since I did not want to go through the hassle of learning to create an “admin plugin” with all the bells and whistles, I decided to write a simple PHP console script to import the XML as mentioned here.

All I needed were two scripts: A parser script to parse the XML output of Disqus comments dump, and secondly, a WordPress handler that loops through these comments and imports them one by one by matching the post’s url attribute and running wp_new_comment() to insert the comment (you can also use the older wp_insert_comment(), but its not the recommended way according to the WordPress codex).

Below is the source code for both these files. First one, console.php is the WordPress handler that you need to run, passing the path of the Disqus comments dump file. And disqus_parse.php is the parser which is called internally by console.php. You need to copy these two files anywhere inside your WP folder structure (I copied them to /wp-content/plugins/test/ folder), and run the console.php from the command line:

console.php:

https://gist.github.com/prahladyeri/e22e4e232416ff841be670601b396c62

disqus_parse.php

https://gist.github.com/prahladyeri/d1e19d8a6d0c7ff23fe3e15f9050b6d3

Running the script

Finally, just keep one thing in mind before running the console.php. Your WordPress system might throw an exception in case it detects too many comments being inserted in a loop. To suppress that exception, you need to add the following line of code to the end of your theme’s functions.php to disable the comment flood filter:

add_filter('comment_flood_filter', '__return_false');

Of course, remember to comment that line once you are done importing the comments.

Update Also check out this recent post which has further fixes and refactoring done to the console.php script

How to install Debian+LXDE on ANY Android Tablet

2017-07-05T00:00:00+00:00

Running a linux distro on android devices is a hot topic these days, and why not? After all, android is already based on linux kernel, but a pretty much locked-down and dumbed-down version of it. The OEM doesn’t give you root and in most cases, not even an open source bootloader or kernel.

That way, its good for maybe the most average user who doesn’t care about the OS and just want to use their phones. But for a power user, that’s not enough.

{.size-full .wp-image-553} Debian (LXDE) running on Xiaomi MiPad Tablet

The so called apps available in the Play Store don’t allow you to utilize the full power of your linux device. A power user always wants to have the power of a full linux distro (such as Debian or Ubuntu or Fedora) on his/her device.

For a long time, I researched for the best way to install a linux distro on a tablet, preferably one that didn’t involve rooting or partitioning the device. The debian guide lists down several methods and apps to do this such as Linux Deploy, GNURoot, Termux, etc. Out of them, the least risky and one that supports most android devices (including my KitKat tablet) is GNURoot-Debian.

Once you install this app on your tablet, it creates a self-contained chroot install of debian using a tool called Proot. Once that is done, all you need to do is open up the app terminal and just start using it:

apt-get update
apt-get install tmux vim gcc python python3 python3-pip

Of course, these are just some of the packages that I’ve installed, you can do whatever you want with your linux installation. The only limitation is regarding multiple user logins. By default, the app will login you as root and while you can create additional users using useradd command, don’t expect things like setuid and setgid to work. The only way to switch user is using the su - yourLogin command (whilst the login command should work too in theory, I’m having a few problems with it presently, it might need some fixing in /etc/pam.d/* configuration files).

Further, if you have good amount of RAM on your tablet, you may consider using a desktop environment along with your headless installation (LXDE is recommended as it performs best on minimal resources). In order to do that, you’ll need two things:

XServer-XSDL app
LXDE/XFCE desktop installed.

The former is used to interact with the headless XServer installation of your debian and provide you a graphical desktop environment. For the latter, you’ll have to just apt-get install lxde (or lxde-core depending on your choice). Then, in order to use the desktop any time on your tablet:

1) Open XServer-XSDL app, follow the instructions until you reach a blue screen.

2) Go to the debian installation and run:
export DISPLAY=:0 PULSE_SERVER=tcp:127.0.0.1:4712
startlxde &

3) Go back to XServer-XSDL app to interact with the desktop.

If all goes well, you should be able to see an lxde desktop like the one shown in the above screen. Coupled with the keyboard case and a bunch of great linux apps (such as vim, emacs, geany, inkscape, eclipse, etc.), you should be able to convert your mobile tablet into a great development machine on wheels.

References:

Flask Recipe - RESTful CRUD using sqlalchemy

2017-06-25T00:00:00+00:00

RESTful apps are a thing these days. When your application’s userbase gets quite large and the client could vary from a laptop to an android device to an iOS device, it pays to keep the backend code separate and use the server only for making RESTful calls using HTTP methods that pertain to basic OLTP transactions: SELECT, INSERT, UPDATE and DELETE.

Popular third-party apps like Firebase essentially provide you this same thing - A REST based front to a database that could be accessed online using simple HTTP methods. But in this tutorial, we will learn how to create such a backend ourselves using Python’s flask framework and sqlalchemy, a light-weight but powerful ORM library that can access ANY database using its flexible sql expression language.

Rather than using firebase, if you develop your own implementation of your back-end, not only will it help you learn and become a better programmer, but also give you a flexible solution that you can scale and change as per your needs. Its also much cheaper to host your own solution on Amazon EC2 (or Lambda) compared to other costlier alternatives.

Contrary to popular thinking, its not very difficult to create a database agnostic backend such as the one represented in the above diagram. With a minimal and powerful web framework such as flask, combined with the power of sqlalchemy, you can get up and running within minutes. In fact, I’ve developed a prototype version called Tiddly that essentially does the same thing as above using just 172 lines of Python code. You can refer to that github repository for reference as we proceed through this tutorial, or directly start using it. But make sure you install the following dependencies before running it:

pip install flask
pip install sqlalchemy

The first step towards creating the app is creating your database models. Once you’ve done the brainstorming and decided what tables and fields you are going to need, you can create a models.py source file with something like this:

import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy import Column, String, Integer
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base

#TODO: Change as needed:
engine = create_engine("sqlite:///tiddly.db", echo=True)
Base = declarative_base()

Session = sessionmaker(bind=engine)
dbsession = Session()

class User(Base):
	__tablename__ = "user"
	id = Column(Integer, primary_key=True)
	email = Column(String)
	password = Column(String)
	name = Column(String)
	def repr(self):
		return "<User(name=%s, email=%s, )>" % (name, email)
		
class Dual(Base):
	__tablename__ = "dual"
	id = Column(Integer, primary_key=True)
	text = Column(String)
	def repr(self):
		return "<Dual(id=%s, text=%s, )>" % (id, text)

I’m using sqlite database for example here, but you can use any one of your choice. A user table is a pretty basic one in almost every app as it is used for authentication. Apart from that, I’ve also created a dual table just to play around with.

After that, create the second file app.py that contains our application code. Define the following import statements along with your models as they will come very handy:

import flask
from flask import request, jsonify, session
import sqlalchemy
from sqlalchemy import inspect, desc
import json
import models
from models import engine, dbsession

Now, the only thing that remains to be done is the plumbing the HTTP methods to their respective database operations. You can either create a separate view function for each one or use a single one for all of them. In this example, I’m using a single function for simplicity.

@app.route("/<table_name>", methods=["POST", "PUT", "DELETE", "FETCH"])
def fetch(table_name):
	print("verb: %s, tablename: %s" % (request.method, table_name))
	if request.method == "POST" or request.method == "PUT":
		data = request.get_json(force=True)
		print("data:", data)
		try:
			TableClass = models.get_class_by_tablename(table_name)
			if TableClass == None: raise Exception("Table not found: %s" % table_name)
			if request.method == "POST": #insert data
				object = TableClass(**data)
				dbsession.add(object)
				dbsession.commit()
			else: #update data
				object = dbsession.query(TableClass).filter_by(**{"id":id}).first()
				if object == None: raise Exception("No data found.")
				#object.update(**data)
				for key in data.keys():
					setattr(object, key, data[key])
				#dbsession.add(object)
				dbsession.commit()
			return jsonify({
				"status": "success",
				"id": object.id,
				})
		except Exception as e:
			return jsonify({
				"status": "error",
				"error": str(e),
				})
	elif request.method == "DELETE":
		try:
			TableClass = models.get_class_by_tablename(table_name)
			if TableClass == None: raise Exception("Table not found: %s" % table_name)
			object = dbsession.query(TableClass).filter_by(**{"id":id}).first()
			if object == None: raise Exception("No data found.")
			dbsession.delete(object)
			dbsession.commit()
			return jsonify({
				"status": "success",
				"id": object.id,
				})
		except Exception as e:
			return jsonify({
				"status": "error",
				"error": str(e),
				})
	elif request.method == "FETCH":
		try:
			data = request.get_json(force=True)
			data = json.loads(data)
			print("data: ", data)
			print("data-type: ", type(data))
			TableClass = models.get_class_by_tablename(table_name)
			if TableClass == None: raise Exception("Table not found: %s" % table_name)
			
			query = dbsession.query(TableClass).filter_by(**data['where'])
			if 'orderby' in data:
				for cname in data['orderby'].split(','):
					reverse = False
					if cname.endswith(' desc'):
						reverse = True
						cname = cname[:-5]
					elif cname.endswith(' asc'):
						cname = cname[:-4]
					print("cname: ", cname)
					column = getattr(TableClass, cname)
					if reverse: column = desc(column)
					query = query.order_by(column)
			if 'limit' in data:
				query = query.limit(data['limit'])
				query = query.offset(data['offset'])
			object = query.all()
			data = [object_as_dict(t) for t in object]
			return jsonify({
				"status": "success", 
				"data": data
				})
		except Exception as e:
			return jsonify({
				"status": "error",
				"error": str(e),
				})
	else:
		return jsonify({
			"status": "error", "error": "Unrecognized verb.",
			})

I’ve used a non-standard HTTP method, FETCH for the SELECT action. That’s because if you use the GET method, you aren’t allowed to actually post data (as in actual posting, don’t confuse with POST method) as per the HTTP specification. The other methods, viz POST, PUT and DELETE are self-apparent and they stand for INSERT, UPDATE and DELETE actions respectively.

As you can see, the app makes good use of the sql expression language of sqlalchemy to dynamically query any kind of data, not only using the usual where clause, but also using ordering and pagination (limit/offset) parameters:

if 'orderby' in data:
	for cname in data['orderby'].split(','):
		reverse = False
		if cname.endswith(' desc'):
			reverse = True
			cname = cname[:-5]
		elif cname.endswith(' asc'):
			cname = cname[:-4]
		print("cname: ", cname)
		column = getattr(TableClass, cname)
		if reverse: column = desc(column)
		query = query.order_by(column)
if 'limit' in data:
	query = query.limit(data['limit'])
	query = query.offset(data['offset'])

The front-end sends whatever it needs to the back-end using JSON format and the result is also in JSON. For example, the following JSON when posted to /user endpoint using FETCH method, returns the record from user table where name field matches admin and orders the results by email in descending order.

{"where": {"name":"admin"}, "orderby": "email desc"}

Adding the limit and offset clauses to the same can help the front-end with pagination.

{"where": {"name":"admin"}, "orderby": "email desc", "limit":2, "offset": 2}

Its also pretty trivial to implement user authentication with this design. I haven’t done it in this example for simplicity, but you can find it in the github code.

All code in this tutorial and on github is MIT licensed and free to use. So, enjoy coding, build your own RESTful CRUD app!

How to create a Google Drive App in PHP

2017-01-02T00:00:00+00:00

This is the second article in the drive series for web programmers that aims to explain how to use the Google Drive API in your web applications to access files/folders on behalf of your logged-in users.

One of the basic tasks here is to authenticate to google and access the drive on the user’s behalf once they grant permission to your app. This method of authentication is called OAuth and is very much needed for implementing the drive api.

However, a good documentation to implement this in a backend app, especially a php app is very much lacking. The so called quickstart for drive api and the web based example here show some example code, but what a lot of beginner programmers need is a step-by-step tutorial of how to go about doing it.

I: Register a google app by visiting the Google API console:

The way the latest version (V3) of drive API works is only through OAuth. It means you cannot put a password or API key inside your code and access the drive files. You need to register your backend app and generate OAuth credentials for the app, so that it can access the drive on the user’s behalf once the user grants permission to the app. So the first step is going to the Google API console, registering the app itself and generating OAuth credentials. The registration process is pretty straightforward, we just select “Create Project” from the dropdown and give a nice name for the project such as Drive Example App in our case.

II: Configure the credentials and download the client_id.json file:

This is the credential file that validates to Google who you are (as a developer) and also your app that acts on your behalf. Download and save it as client_id.json in the same directory as your app.

III: Write your back-end app:

First of all, you have to add the dependency of google-api php library to your project. If you are using composer, all you need to do is add this package to the composer.json:

"require": {
  "google/apiclient": "^2.0"
}

If you don’t use composer, you can just download the latest version of library from their repo, and just require_once it in your code like this:

require_once '/path/to/google-api-php-client/vendor/autoload.php';

You can follow this pattern for any kind of php project, be it based on Symfony, Laravel, CodeIgniter or even a pure php project. But this tutorial and code example is based on a pure php project.

The first thing to do now is to handle the home page url (index.php).

$client = new Google_Client();
$client->setAuthConfig('client_id.json');
$client->addScope(Google_Service_Drive::DRIVE);

if (file_exists("credentials.json")) {
	$access_token = (file_get_contents("credentials.json"));
	$client->setAccessToken($access_token);
	//Refresh the token if it's expired.
	if ($client->isAccessTokenExpired()) {
		$client->fetchAccessTokenWithRefreshToken($client->getRefreshToken());
		file_put_contents($credentialsPath, json_encode($client->getAccessToken()));
	}
	$drive_service = new Google_Service_Drive($client);
	$files_list = $drive_service->files->listFiles(array())->getFiles(); 
	echo json_encode($files_list);
} else {
  $redirect_uri = 'http://' . $_SERVER['HTTP_HOST'] . '/oauth2callback.php';
  header('Location: ' . filter_var($redirect_uri, FILTER_SANITIZE_URL));
} We first check whether we have the drive access credentials for the user locally stored in a file called credentials.json (not to be confused with client\_id.json we downloaded earlier which is for developer credentials). Again, we are assuming a single user scenario here. If your drive app needs to authenticate with multiple users, you’ll have to store separate credentials.json for each logged-in user in the database, and access that through a session or something.

Further, if credentials aren’t found locally, we direct them to /oauth2callback.php, so google will authenticate them and send us the token for accessing the drive, and after that, we will put that token into the local file, credentials.json and redirect the user back to the index.php. Finally, we call the listFiles() method that displays the list of all files and folders in that user’s drive. Here is the code for oauth2callback.php:

$client = new Google_Client();
$client->setAuthConfigFile('client_id.json');
$client->setRedirectUri('http://' . $_SERVER['HTTP_HOST'] . '/oauth2callback.php');
$client->addScope(Google_Service_Drive::DRIVE); //::DRIVE_METADATA_READONLY

if (! isset($_GET['code'])) {
  $auth_url = $client->createAuthUrl();
  header('Location: ' . filter_var($auth_url, FILTER_SANITIZE_URL));
} else {
  $client->authenticate($_GET['code']);
  $access_token = $client->getAccessToken();
  file_put_contents("credentials.json", json_encode($access_token));
   
  $redirect_uri = 'http://' . $_SERVER['HTTP_HOST'] . '/';
  header('Location: ' . filter_var($redirect_uri, FILTER_SANITIZE_URL));
}

Once you have the credentials locally (in the form of credentials.json), you can just use it to access the drive API. Thus, the result of this whole exercise is that only on first page load is the user redirected to google site to authenticate themselves. Once the app has the access token (credentials.json), its no longer required, the drive can be accessed directly by the app from then on. If all goes well, you should be able to see a screen such as this when you test this example app for the first time:

I’ll leave the more comprehensive use of this API as an exercise to the reader who wants to develop a more fully featured app out of this. Click the below link to download the source for this example implementation from the Github repo:

php_drive_example{.btn .btn-md .btn-success}

Note:

If you are getting an SSL certificate error while testing this on Windows, have a look at this.

How to create a Google Drive App in Flask

2016-12-29T00:00:00+00:00

This is the first in a series of articles for web programmers that explain in detail about using the Google Drive API in your web applications to access files/folders on behalf of the users of your application. In my last project, I had to develop a python flask app for my users that required to access the files stored in their google drive account.

The major challenge for me was to authenticate to google and access the drive on the user’s behalf once they grant permission to my app. This method of authentication is called OAuth and is very much needed for implementing the drive api.

However, a good documentation to implement this in a backend app, especially a python web-based app is very much lacking. The so called quickstart for drive api shows some example code, but what I needed was a step-by-step tutorial of how to go about doing it. Since I couldn’t find any such tutorial online, I thought about writing one myself.

I: Register a google app by visiting the Google API console:

The way the latest version (V3) of drive API works is only through OAuth. It means you cannot put a password or API key inside your code and access the drive files. You need to register your backend app and generate OAuth credentials for the app, so that it can access the drive on the user’s behalf once the user grants permission to the app. So the first step is going to the Google API console, registering the app itself and generating OAuth credentials. The registration process is pretty straightforward, you just select “Create Project” from the dropdown and give a nice name for your project such as Flask Drive Example App in our case.

II: Configure the credentials and download the client_id.json file:

III: Write your back-end app:

The most important thing to know before building your app is to install these dependencies:

pip install flask google-api-python-client

You can replace flask with django, pylons or any other framework you use, but this tutorial and code example is based on flask. The principle of accessing the drive api should still apply, so you should be able to make use of this code.

The first thing to do is create a flask object and handle the home page url. It could in fact be any other url in your app, but in this example, I’ve used the home page url (/) to do the OAuth authentication on behalf of the logged in user.

app = flask.Flask(__name__)

@app.route('/')
def index():
    credentials = get_credentials()
    if credentials == False:
        return flask.redirect(flask.url_for('oauth2callback'))
    elif credentials.access_token_expired:
        return flask.redirect(flask.url_for('oauth2callback'))
    else:
        print('now calling fetch')
        all_files = fetch("'root' in parents and mimeType = 'application/vnd.google-apps.folder'", sort='modifiedTime desc')
        s = ""
        for file in all_files:
            s += "%s, %s<br>" % (file['name'],file['id'])
        return s

We first check whether we have the drive access credentials for the user locally stored in a file. This is done by the get_credentials() function that checks the local access token file credentials.json (not to be confused with client_id.json we downloaded earlier which is for developer credentials). Again, we are assuming a single user scenario here. If your drive app needs to authenticate with multiple users, you’ll have to store separate credentials.json for each logged-in user in the database, and access that through a session or something.

Further, if credentials aren’t found locally or have expired, we direct them to /oauth2callback, so google will authenticate them and send us the token for accessing the drive, post which, we will put that token into the local file, credentials.json and redirect the user back to this index site. Finally, if the credentials are valid, we call the fetch() function that displays the list of all root folders in that user’s drive along with their IDs. Here is the code for oauth2callback:

@app.route('/oauth2callback')
def oauth2callback():
    flow = client.flow_from_clientsecrets('client_id.json',
            scope='https://www.googleapis.com/auth/drive',
            redirect_uri=flask.url_for('oauth2callback', _external=True)) # access drive api using developer credentials
    flow.params['include_granted_scopes'] = 'true'
    if 'code' not in flask.request.args:
        auth_uri = flow.step1_get_authorize_url()
        return flask.redirect(auth_uri)
    else:
        auth_code = flask.request.args.get('code')
        credentials = flow.step2_exchange(auth_code)
        open('credentials.json','w').write(credentials.to_json()) # write access token to credentials.json locally
        return flask.redirect(flask.url_for('index'))

Once you have the credentials locally (in the form of credentials.json), you can just use it to access the drive API. Thus, the result of this whole exercise is that only on first page load is the user redirected to google site to authenticate themselves. Once the app has the access token (credentials.json), its no longer required, the result is displayed directly on the page from then on. If all goes well, you should be able to see a screen such as this when you test this example app for the first time:

I’ve also included the functions to download and upload files to the drive as they will be very useful:

def download_file(file_id, output_file):
    credentials = get_credentials()
    http = credentials.authorize(httplib2.Http())
    service = discovery.build('drive', 'v3', http=http)
    #file_id = '0BwwA4oUTeiV1UVNwOHItT0xfa2M'
    request = service.files().export_media(fileId=file_id,mimeType='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
    #request = service.files().get_media(fileId=file_id)
    
    fh = open(output_file,'wb') #io.BytesIO()
    downloader = MediaIoBaseDownload(fh, request)
    done = False
    while done is False:
        status, done = downloader.next_chunk()
        #print ("Download %d%%." % int(status.progress() * 100))
    fh.close()
    #return fh
    
def update_file(file_id, local_file):
    credentials = get_credentials()
    http = credentials.authorize(httplib2.Http())
    service = discovery.build('drive', 'v3', http=http)
    # First retrieve the file from the API.
    file = service.files().get(fileId=file_id).execute()
    # File's new content.
    media_body = MediaFileUpload(local_file, resumable=True)
    # Send the request to the API.
    updated_file = service.files().update(
        fileId=file_id,
        #body=file,
        #newRevision=True,
        media_body=media_body).execute()

I’ll leave the more comprehensive use of these functions as an exercise to the reader who wants to develop a more fully featured app out of this. Click the below link to download the flask_drive_example.py script for this example implementation from the Github gist:

flask_drive_example.py

How to create a screencast in Xubuntu using ffmpeg

2016-07-19T00:00:00+00:00

In this article, I’m going to describe how to create a desktop screencast recording with or without audio in Xubuntu 16.04. With little changes, however, it should work on other distros too.

The first thing to do is to ensure that your headphone microphone is enabled in Xubuntu audio settings. For some reasons, Xubuntu doesn’t seem to do this by default. Just click that tray icon top-right corner and click on Sound Settings. Then click on Input Devices tab and select Headset Microphone or something from the list, and unmute the device by clicking the grey color audio icon:

And of course, you’ll need ffmpeg to do this, its a small command line audio utility in linux. Its only a sudo apt-get away in case you don’t have it already. After that, all you have to do is run this command:

ffmpeg -video_size 1024x767 -framerate 25 -f x11grab -i :0.0+0,0 -f alsa -ac 2 -i default output.mkv

To stop the screen recording, just press Ctrl+C. This command uses the alsa module to record audio. The -ac 2 parameter specifies the audio channel (stereo as opposed to mono). If you want, you can also specify an audio sampling rate using -ar option which is usually 44100 by default. The -i default refers to the default available audio device. It usually works, but in case it doesn’t, you can specify the exact input device you want to use. In order to do that, run this command:

arecord -L

And you will get an output such as this one:

dmix:CARD=PCH,DEV=0
	HDA Intel PCH, ALC3234 Analog
	Direct sample mixing device
dsnoop:CARD=PCH,DEV=0
	HDA Intel PCH, ALC3234 Analog
	Direct sample snooping device
hw:CARD=PCH,DEV=0
	HDA Intel PCH, ALC3234 Analog
	Direct hardware device without any conversions
plughw:CARD=PCH,DEV=0
	HDA Intel PCH, ALC3234 Analog
	Hardware device with all software conversions

In the above, you can see the plugged-in available sound card at the end. So you can take that device and specify it in this way instead of the default device:

ffmpeg -video_size 1024x767 -framerate 25 -f x11grab -i :0.0+0,0 -f alsa -ac 2 -i plughw:CARD=PCH,DEV=0 output.mkv

If you don’t want audio in the screencast, however, then omit the alsa module and just run this instead:

ffmpeg -video_size 1024x767 -framerate 25 -f x11grab -i :0.0+0,0 output.mkv

I’ve used other screencasting software in Linux before such as the RecordMyDesktop tool and the built-in Ctrl+Shift+Alt+R screencasting in GNOME. But this method of using ffmpeg was what I found to be the most speedy and configurable. Try it and do let me know how your screencasting goes!

How to Unbrick and revive a totally lifeless Xiaomi Mi-Pad

2016-07-13T00:00:00+00:00

Last week, my brand new Xiaomi Mi-Pad arrived via Flipkart and no sooner did I have it in my hands, I started installing useful apps like Greenify, ES File Explorer, etc. Two days went by and I was disappointed to learn that it doesn’t support a USB Dongle (Data Card) which is a very common method of Internet access in India. The only way I could have it was to root the tablet. And it was while rooting using SuperSU app that I foolishly selected a wrong click and ended up bricking my brand new tablet.

The Problem

It was a hard brick, meaning my tablet showed no signs of life when switched on or connected to a PC through a USB cable. Had it been a soft brick, meaning that I were able to boot to MIUI Recovery by pressing Power+UpVolume or to FastBoot Recovery by pressing Power+DnVolume button, it would have been easy to recover. But my tablet was totally dead. Even after charging for next 6 hours, all it did was light up a black LED screen for a second or two and then vanished for any combination of those buttons.

The problem was that the wrong click that I made had caused the tablet bootloader to be corrupted. And in the android world, if the bootloader is lost, then your device is literally as useful as a brick! Unlike 8086 PCs, android tablets don’t have a hard-wired BIOS to recover things from, the bootloader is as good as the BIOS.

The Research

Now obviously, I was disappointed and annoyed. After spending ₹10,999 on a brand new tablet, this was not what I was hoping for. Getting service from a Xiaomi service center was not an option as they are very rare in India. Besides, OEMs hardly provide warranty/support to people who try and root their devices, do they? So naturally, I had to myself find a way out of this. My research finally led me to a post by a Russian hacker who had successfully revived the tablet from this very situation. In this article, I’ll note down exactly what steps I had taken to revive it and hope that someone might find it useful.

The Solution

Though the solution is easy and straight-forward, there are some tricky areas where you might run into trouble. If you don’t have the confidence to open up an electronic device using a screwdriver or flash a stock ROM, just say quits and if possible, ask help from someone you know who is used to do it. Basically, the process involves this:

Open the back-cover by separating it from the screen (fingernails or a sharp object will do) and unscrew the plastic panel that covers the top area of the motherboard.
Disconnect the battery by slowly pulling the power belt (using a flat-head screwdriver or any sharp object).
Connect the USB cable to Mi-Pad end (but not the PC end).
Create a short circuit between the two tiny pads inside the MicroSD card slot using a metal object or wire.
Connect the USB cable to the PC end.

If done properly, the PC will detect a Nvidia Boot Recovery for Mobile USB device and then you can flash your stock ROM using Xiaomi’s flashing software, Mi Flash Tool, thus putting the life back into your tablet.

The Magic

The most tricky and magical aspect of this whole process is the short-circuiting in step four. Firstly, it is not easily diagnosable. The Nvidia USB driver is in fact detected by the PC even if you don’t create the short-circuit (but the battery needs to be disconnected of course), but then it will disappear instantly and you won’t get the time to flash your stock ROM. For some unknown laws of electricity (or maybe the laws coded inside this Nvidia driver), the short-circuit may not work instantly the first time. You’ll create the short-circuit, connect to the USB port and hear the Windows beep sound for USB connection, but moments later you will hear another beep of disconnection. If that happens, you’ll have to disconnect the USB cable and repeat from step-4: Create the short-circuit again and reconnect to the USB port - I had to do this about 50 times before the connection was stable enough for me to flash the ROM.

Some forum posts advice you to plug in the power belt at exactly the same time as the USB connection is detected in the PC. However, I’ve found that its not true. Not once did I put the power belt back in the whole process until the ROM had been flashed completely.

The Weapon

You can use any metallic object to create the short circuit, even a tiny wire could be used. The important thing is that those two minuscule pads inside the MicroSD slot are connected by a metal. Personally, I used this tiny pin that came with the booklet along with the Mi-Pad itself!

{.size-large}

All I did was place that pin inside the MicroSD slot, so it created a bridge on top of those two pads, and supported the pin using any available object (in this case a screwdriver).

You’ll also need a 2.0mm screwdriver to open the back-panel above the motherboard. Please remember that opening those screws are going to void your warranty (which is going to happen in any case since you either tried to root or used the dd command to rewrite partitions in order to reach this stage!).

On your Windows PC, you need the following installed:

Latest version of Mi Flash Tool (Can be downloaded here).
Latest version of Mi Phone Manager PC Suite for the Nvidia drivers (Can be downloaded here).

Alternatively, you can download the whole bundled software from here that comes with Mi Flash Tool, NVidia Drivers and ADB Drivers too.

You’ll also need to download the latest MIUI stock ROM for Mi-Pad from here and extract somewhere on your PC to flash it later.

Once your short-circuit works and the connection stays stable, waste no more time. Just fire up the Mi Flash tool and select the path to the ROM image you just extracted and click the flash button. Once the flash is successful, assemble back your Mi-Pad and on pressing the power button, you should see this:

Summary

Keep a cool mind and do it, it can be done! If you can’t do it, try and find a mobile repair shop nearby or catch a tech enthusiast in your circle who can do it.

References

How to deal with frequent disconnects of 3G USB Dongle on Linux Desktop

2016-06-28T00:00:00+00:00

One of the major issues on linux desktop these days has to do with 3G USB dongles/modems. In many countries like India, a USB dongle still remains a major way to access the Internet. The latest versions of network-manager has solved most of the issues relating to the basic recognition of these modems, but one major irritant still remains on Ubuntu 16.04 and a few other distros.

It so happens that many a times, the connection entirely vanishes from the network-manager applet menu on the top right. For example, if you hibernate your laptop and then wake it, the menu is no longer there. However, if you manually take out the dongle and re-insert, the modem is recognized and the menu reappears. If you want to avoid that manual re-insertion and want to do a “soft reset” instead, here is what you can do:

Send a reset signal to your usb modem:
```
sudo usb_modeswitch -v 12d1 -p 1496 -R
```
Send a modeswitch signal to your usb modem:
```
sudo usb_modeswitch -v 12d1 -p 1496 -J
```

The parameters, -v and -p stand for vendor code and product code respectively and you can find them by running lsusb command. For example, the below entry is for vendor code 12d1 (Huawei) and the product code is 1496.

Bus 001 Device 004: ID 12d1:1496 Huawei Technologies Co., Ltd. Broadband stick

Please remember that for modeswitch signal, the parameter -J is vendor specific and works for Huawei modems only, please run man usb-modeswitch to find out your vendor specific modeswitch control parameter. For instance, its -O for Sony and -L for Cisco control parameter.

Of course, once you try these commands and find that they are working, you can add them as a shortcut to your ~/.bashrc file. Something like this:

alias "reset_huawei=sudo usb_modeswitch -v 12d1 -p 1496 -R"
alias "modeswitch_huawei=sudo usb_modeswitch -v 12d1 -p 1496 -J"

Now, all you need to do is run reset_huawei && modeswitch_huawei and that’s it! No need to manually re-insert the dongle anymore.

List of free online programming courses (MOOC) offering FREE certificates of accomplishment

2016-06-07T00:00:00+00:00

This post is inspired by the Coursera Junkie post by Kathryn last year that happened to invite the ire of lots of learners interested in MOOC (Massive Online Open Courses). Whilst I was observing the comment chain in that post, I had never bothered to actually take the time to research any MOOC courses that offered free certificates or statements of accomplishment. But a few months ago, my once favorite online academy, edX.org had just crossed a line by starting to charge money for their certificates, thus shutting down the last zion of open education left for the learners. And because of this, I felt the need to start researching the alternatives.

Now, I know that lots of critics would be throwing this “free rider” argument against me saying that why should I bother about a few dollars worth if I’m getting a verified certificate, or that paying for a certificate somehow increases the “value” of the course, etc. However, you fail to understand that ultimately, the question is not about the certificate (which is just a piece of paper, anyways). It’s really about the learnings and insights about a subject that you take away from a MOOC course. It’s also about the collaboration and interaction with other co-students on the academy who take the course along with you. The PDF certificate or transcript is just a downloadable thing that you use to improve your LinkedIn profile, but more importantly, it’s an acknowledgement by the academy that you underwent that course.

Now, as an online academy, if you are charging money to give us that acknowledgement, you are no academy but just a profiteer who is running a business. The moment that money comes in the picture, an academy turns into a business and its education quality starts to deteriorate. Right to education, of all things, MUST be based on merit. You can’t bar someone’s entry to education based on the financial ability. This same concept of merit also applies to Open Source software development. Every contributing developer is invited to the project on the basis of merit alone. Further, every commit, every feature request and every decision is based on merit and voting, no company can buy its way through a FOSS project, the Linux kernel project is the proof of that!

If Coursera and edX stopped being academies, and turned into profit-making entities, let them be - there is no dearth of online learning resources! Below are some MOOC courses that provide free transcripts or certificate of accomplishments upon completion (In future, if any of these academies turn out to be like Coursera/edX, rest assured, they will be removed from this list!):

CS101: Introduction to Computer Science I via Saylor Academy
CS102: Introduction to Computer Science II via Saylor Academy
CS107: C++ Programming via Saylor Academy
CS201: Elementary Data Structures via Saylor Academy
CS202: Discrete Structures via Saylor Academy
CS301: Computer Architecture via Saylor Academy
CS302: Software Engineering via Saylor Academy
CS303: Algorithms via Saylor Academy
CS304: Compilers via Saylor Academy
CS305 Web Development via Saylor Academy
CS401: Operating Systems via Saylor Academy
CS402: Computer Communications and Networks
CS403: Introduction to Modern Database Systems
CS404: Programming Languages via Saylor Academy
CS405: Artificial Intelligence via Saylor Academy
CS406: Information Security via Saylor Academy
CS408: Advanced Artificial Intelligence
CS409: Cryptography via Saylor Academy
CS410: Advanced Databases via Saylor Academy
CS412: Mobile Applications Development via Saylor Academy
Learn HTML5 Programming From Scratch via Udemy
Courses from Open Universities, Australia via open.edu.au
Concepts in Game Development via open2study.com
Compiler Design Tutorial: What is, Types, Tools, Example via guru99.com

Provision of free certificate transcripts/statement of accomplishments in each Academy:

=> Saylor Academy provides free certificate of completion upon passing the final exam of every course at learn.saylor.org:

A free certificate of completion is available for every course at learn.saylor.org by passing a course’s final exam with 70% or better. Your digital certificate can be shared privately or publicly, linked to from other websites, downloaded to a PDF file, and printed.

=> Udemy offers certificate of completion for many of its courses

Most courses offer a certificate of completion. When all lectures have been completed, a gold or green trophy will appear, signifying that the certificate of completion is ready for download. Click on the trophy to view the certificate.

=> OpenupED has multiple recognition options. Apart from a free badge/certificate, some courses also offer official formal certificate by the university.:

Always a badge and/or a certificate. I.e. you have the opportunity to get a badge or a certificate of course completion for free (as evidence of completion).

FormalCertificateIn addition the majority of OpenupEd MOOCs provide the possibility to obtain a formal certificate

=> Stanford.edu offers a statement of accomplishments for some of their “In Session” courses:

In Session courses offer a Statement of Accomplishment issued by the instructor upon successful completion.

=> open2study.com offers a free downloadable certificate of accomplishment upon course completion:

Once the course finishes, you will be able to download a certificate of achievement, along with your final grades, via My Study Centre.

Ten useful LibreOffice Macro Recipes

2016-02-26T00:00:00+00:00

Macros are a great way to automate tasks in Spreadsheet applications, be it the good old Microsoft Excel or the equally efficient FOSS alternative, LibreOffice Calc. The best thing about macros is that they are often written in a very easy to grasp language called Basic (though Calc also supports Python these days).

As its very name suggests, Basic is a lenient programming language actually designed with ease of use in mind. For instance, upper/lower case doesn’t matter for variable names or keywords (if/IF, sub/Sub, function/Function are equivalents), function braces are optional like Ruby and type-conversion happens automatically. This makes Basic highly useful for both power users and programmers. A LibreOffice Basic macro is just a function or sub procedure which does a specific useful task. In this tutorial, we will see ten such useful macros which can help you with various automation tasks.

Recipe 0: How to create a LibreOffice macro
Recipe 1: Read cell contents
Recipe 2: Change cell contents
Recipe 3: Search and Replace text
Recipe 4: Regular Expressions
Recipe 5: Show File-open dialog
Recipe 6: Show File-save dialog
Recipe 7: File I/O: Read from files
Recipe 8: File I/O: Write to files
Recipe 9: Load data from a CSV file
Recipe 10: Copy text to clipboard
Demo
Further Reading

Recipe 0: How to create a LibreOffice macro

Whilst macros can be created in Writer and Draw too, in this specific tutorial, we will restrict ourselves to spreadsheets (Calc). To create a macro, just open the spreadsheet in LibreOffice and go to Tools->Macros->Organize Macros->LibreOffice Basic menu. After that, if you want to create a macro specific to your spreadsheet (as usually is the case), expand your spreadsheet file on left and select Standard and click New. This will open the LibreOffice Macro Editor as a separate window.

Recipe 1: Read cell contents

One of the most basic things needed for automation is reading a cell’s contents. The following piece of code does exactly this:

Sub read_cell
	dim document as object
	document = ThisComponent
	sheet = document.Sheets(0)
	MsgBox(sheet.getCellByPosition(0, 0).String)
End Sub

dim is a keyword used to declare a variable but declaration is totally optional unless Option Explicit is specified at the beginning of the module. ThisComponent is the LibreOffice object that references the current spreadsheet (or a written document in case of Writer). The important thing here is the expression, sheet.getCellByPosition(0, 0).String which gets the contents of first cell in the first row. Cells can be referenced using the co-ordinate system where (0,0) refers to cell at row-0 and column-0. Thus, any value across the entire spreadsheet can be fetched using this simple method.

To run a macro from the editor, just place the cursor inside the sub or function body of any macro and press F5 (or alternatively, click the Run BASIC icon on the toolbar).

Recipe 2: Change cell contents

Another often needed thing is the ability to change the cell contents. The following code sets the first cell in the first row to “Hola! Mundo”, the Spanish expression for “Hello! World”:

Sub change_cell
	dim document as object
	document = ThisComponent
	sheet = document.Sheets(0)
	sheet.getCellByPosition(0, 0).String = "Hola Mundo!"
	MsgBox("Done")
End Sub

Recipe 3: Search and Replace

Searching and replacing specific strings could be an important part of your automation routine. Below is a fun macro that searches for the first names of some Linux experts (like Linus, Richard, Peter, etc.) and replaces it with their last names (Torvalds, Stallman, Anvin):

Sub replace_text
	Dim names() As String
	Dim surnames() As String
	Dim n As Long
	Dim document As Object
	Dim sheets as Object
	Dim sheet as Object
	Dim replace As Object

	names() = Array("Linus", "Richard", "Peter", "Greg", "Bill")
	surnames() = Array("Torvalds", "Stallman", "Anvin", "Kroah", "Gates")
	document = ThisComponent rem .CurrentController.Frame
	rem sheet = doc.CurrentSelection.Spreadsheet
	sheets = document.getSheets()
	sheet = sheets.getByIndex(0)
	replace = sheet.createReplaceDescriptor rem document.createReplaceDescriptor in case of Writer
	rem replace.SearchRegularExpression = True
	For n = lbound(names()) To ubound(names())
		replace.SearchString = names(n)
		replace.ReplaceString = surnames(n)
		sheet.replaceAll(replace)
	Next n
	MsgBox("Done")
End Sub

names() and surnames() are actually arrays. Unlike C and Java, arrays in Basic are declared and accessed using round braces and not square ones. Also, what gets declared in an array declaration is the upper-bound, not the total size. Thus, foo(2) is actually a size-3 array ranging from indices 0 to 2.

Recipe 4: Regular Expressions

Regular expressions are very useful in searching and replacing text based on specific patterns. The following macro searches for all the email addresses in your spreadsheet and replaces each one with foo@bar.com:

Sub replace_with_regex
  Dim names() As String
  Dim surnames() As String
  Dim n As Long
  Dim document As Object
  Dim sheets as Object
  Dim sheet as Object
  Dim replace As Object

  pattern = "\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b" rem regex pattern to match any email address
  document = ThisComponent rem .CurrentController.Frame
  sheets = document.getSheets()
  sheet = sheets.getByIndex(0)
  replace = sheet.createReplaceDescriptor rem document.createReplaceDescriptor in case of Writer
  replace.SearchRegularExpression = True
  replace.SearchString = pattern
  replace.ReplaceString = "foo@bar.com"
  sheet.replaceAll(replace)

  MsgBox("Done")
End Sub

Recipe 5: Show File-open dialog

Showing the File-open dialog is a very common requirement, especially when you want to open an external file for processing. The below code uses the FilePicker object to show the file-open dialog and return the selected file-name:

function show_open_dialog
	dim aurl as object
	dim s as string
	dim oDlg as object
	
	oDlg = createUnoService("com.sun.star.ui.dialogs.FilePicker")
	oDlg.setMultiSelectionMode(false)
	oDlg.appendFilter("CSV Files (.csv)", "*.csv" ) 
	oDlg.execute
	aUrl = oDlg.getFiles()
	
	s = aUrl(0)
	if len(s) > 0 then
		MsgBox("File Selected: " & s & chr(13))
	end if
	show_open_dialog = s
end function

createUnoService is a LibreOffice specific method for creating helper objects like FilePicker in this example. The appendFilter method is used to filter out only CSV files in the dialog.

Recipe 6: Show File-save dialog

For showing a File-save dialog, the same FilePicker object is used, initializing it with the FILESAVE_AUTOEXTENSION argument:

function show_save_dialog
	dim aurl as object
	dim s as string
	dim oDlg as object
	
	sFilePickerArgs = Array(_
	com.sun.star.ui.dialogs.TemplateDescription.FILESAVE_AUTOEXTENSION )    
	oDlg = createUnoService("com.sun.star.ui.dialogs.FilePicker")
	oDlg.initialize(sFilePickerArgs())
	oDlg.setMultiSelectionMode(false)
	oDlg.appendFilter("CSV Files (.csv)", "*.csv" ) 
	oDlg.setTitle("Save As....")
	
	if oDlg.execute() then
		aUrl = oDlg.getFiles()
		s = aUrl(0)
		if len(s) > 0 then
			MsgBox("File Selected: " & s & chr(13))
		end if
	else
		s = ""
	end if
	show_save_dialog = s
end function

Recipe 7: File I/O: Read from files

Raw file I/O is a feature provided by almost every language and Basic macros make it almost too easy. Below code is used to read a CSV file with three columns. Name of the file is set in the filename variable. The variable num is a numerical tag used to reference the file-handler and FreeFile() returns a free available number that can be used for tagging. The open statement is self-explanatory. In Basic, files can be opened in Input, Output and Binary modes. Finally, the input statement is used to actually read the file into the variables line after line.

sub file_io_read
	dim v1, v2, v3
	filename = "/home/prahlad/data/test.csv"
	num = FreeFile()
	open filename for input as #num 
	do while not eof(num)
		input #num, v1, v2, v3
		print v1 & "::" & v2 & "::" & v3
	loop
	close #num
	msgbox "Done"
end sub

Recipe 8: File I/O: Write to files

For writing to files, a handler is opened in output mode instead of input, and the write statement is used to actually write the variables to a file.

sub file_io_write
	filename = "/home/prahlad/data/dummy.csv"
	num = FreeFile()
	open filename for output as #num 
	write #num, "col1", "col2", "col3"
	write #num, "1", "2", "3"
	write #num, "4", "5", "6"
	close #num
	msgbox "Done"
end sub

Recipe 9: Load data from a CSV file

Apart from working in raw I/O mode, it is sometimes required to load a complete CSV as a sheet in the current document. Using the show_open_dialog function that we studied earlier, the following macro first prompts a user with a File-open dialog and then loads the specified CSV file as a new sheet:

sub load_from_csv
	fname = show_open_dialog
	if len(fname)>0 then
		dim fileProps(1) as new com.sun.star.beans.PropertyValue
		fileProps(0).Name = "FilterName"
		fileProps(0).Value = "Text - txt - csv (StarCalc)"
		fileProps(1).Name = "FilterOptions"
		fileProps(1).Value = "44,34,76,1,,0,false,true,true,false"
		document = StarDesktop.loadComponentFromURL(fname, "_blank", 0, fileProps())        
	end if
	msgbox "Done"
end sub

fileProps(0) is a property variable used for specifying the CSV file format, while fileProps(1) specifies the default formatting options for the CSV (such as a delimiter, charset, etc.)

Recipe 10: Copy text to clipboard

Your custom processing might involve putting a specific text to the clipboard from LibreOffice Calc. Following code shows how to put the string “Hola!” to the system clipboard:

sub copy_to_clipboard
	oClip = CreateUnoService("com.sun.star.datatransfer.clipboard.SystemClipboard")
	oTR = createUnoListener("TR_", "com.sun.star.datatransfer.XTransferable")
	oClip.setContents(oTR, null)
	msgbox "Done"
end sub

Function TR_getTransferData( aFlavor As com.sun.star.datatransfer.DataFlavor ) As Any
	If (aFlavor.MimeType = "text/plain;charset=utf-16") Then
		TR_getTransferData = "Hola!"
	EndIf
End Function

Function TR_getTransferDataFlavors() As Any
	Dim aF As new com.sun.star.datatransfer.DataFlavor
	aF.MimeType = "text/plain;charset=utf-16"
	aF.HumanPresentableName = "Unicode-Text"
	TR_getTransferDataFlavors = Array(aF)
End Function

Function TR_isDataFlavorSupported( aFlavor As com.sun.star.datatransfer.DataFlavor ) As Boolean
	TR_isDataFlavorSupported = (aFlavor.MimeType = "text/plain;charset=utf-16")
End Function

Second function is a callback and is used for storing the string to clipboard. The last two are helper functions used by the SystemClipboard and XTransferable helper objects and are required.

Demo

Finally, the working LibreOffice Calc spreadsheet implementing all these examples can be found here.

Further Reading:

How to assemble a fast and minimal Debian Desktop using Openbox

2016-02-16T00:00:00+00:00

Being a web developer, one of the nagging things to do is keeping your hardware in sync with your performance requirements. In the good old times, a P4 or even a Celeron based PC or laptop was enough for programming. But with changing times, the required investment to get a sane performance has increased to astronomical levels.

Problem is not just with the limitations of hardware, but our requirements too have increased. Apart from the tons of most needed apps like eclipse, firefox, libreoffice, thunderbird, etc., you now have to keep VMs running in virtualbox at the same time you are having a chat on skype with your client. Add to that, heavy things like eclipse or Android Emulator and your linux (or even Windows) desktop will start to buckle under the pressure.

All in all, even an Intel i5 or i7 chip isn’t sufficient today to handle multiple heavy or memory consuming apps. The only solution I have found is to use a lighter window-manager like Openbox, Fluxbox or IceWM instead of a heavy Desktop Environment like GNOME, Unity or KDE. This tutorial contains all the steps needed to assemble on your own light-weight DE based on Openbox on Debian (or one of its compatible derivatives like Ubuntu or Linux Mint).

{.size-full .wp-image-498 width=”956” height=”537”} Openbox

Openbox is an open source project (GPL licensed) based on the good old blackbox. It provides just the minimal framework to build a no-flashy, sane desktop environment that gets out of your way, so most resources are used for running the actual apps that you use. Openbox occupies less than 8 MB of RAM on your machine. Again, this kind of setup is not for everyone, in case you are a huge fan of the Unity dash or GNOME dash, you may not like Openbox at all. On the other hand, if you like the new MATE Desktop which is based on the old GNOME 2, you may find this approach interesting. In fact, Openbox is the window manager used by LXDE, the DE most famous for being light on resources. You are, in fact, going to create something like your own version of LXDE by following this tutorial, only its even more minimal and custom to your requirements. Here is how to do it:

Download the Debian minimal net-install ISO from here. It is roughly 150-200 MB in size.
Either burn the ISO to a USB drive or try it out in a VM first.
Use the text based or graphical installer to complete the installation.

Once you land on the command line, login as the superuser (su) and install these packages:

apt-get install network-manager
apt-get install xorg openbox xdm
apt-get install xbacklight pcmanfm lxappearance lxpanel gmrun gnome-terminal

To make the panel automatically show up, add this to ~/.config/openbox/autostart file using nano or vim editors:
```
lxpanel &
```
Restart

Note that this will create only a bare minimum, workable DE with just a File Manager and no other usable apps. lxpanel is a “Windows-98 style” panel that sits on top/bottom of your desktop and gives a “start-menu” in which your programs are organized. lxappearance is a handy tool to switch GTK themes which comes very useful. You still have to install evince to read PDF books, geany or gedit for a GUI text editor, iceweasel and icedove for a browser and mail client respectively, and libreoffice-calc for a spreadsheet.

But the good thing is that you are the master of your desktop world now. You don’t have to live with that extra load of bloated baggage that heavy DEs usually come with. If you follow the above process correctly, you will end up with an installation size of less than 500 MB! And your own custom-made DE that works!

As for customization, Openbox is highly customizable. Arch Linux also has a wonderful documentation on this topic, and here is another brief guide. For menus, you can either install obmenu or if you are a simplicity fan like me, you can just edit your way through ~/.config/openbox/menu.xml. This, and the other file, ~/.config/openbox/rc.xml are the only two Openbox configuration files and they have all the needed options. In fact, if you decide to make use of the lxpanel main menu, you don’t even have to edit the menu.xml of Openbox.

Notes:

lxpanel is not the only option for installing a Desktop panel, there are others too like fbpanel, tint2 and xfce4-panel. Read this for more details.
To add a battery indicator to the lxpanel, add this to the ~/.config/lxpanel/default/panels/panel
```
Plugin {
  type = batt
}
```
To save your openbox customizations between sessions (which is a basic requirement), copy the rc.xml and menu.xml from /etc/X11/openbox to ~/.config/openbox/ (create this folder if it doesn’t exist).
gmrun is needed to show the run dialog that pops up when you press Alt+F2 in any “normal” DE. Once you install gmrun, just add below code to ~/.config/openbox/rc.xml somewhere in the <keyboard> section, in order to make the keys work:
```

<keybind key="A-F2">
<action name="Execute">
  <command>gmrun</command>
</action>
</keybind>

```
xdm is optional, but helpful if you don’t want to do a lot of configuration changes in xorg files to show up the desktop automatically each time you login. Of course, you can use lightdm too if you want.
I don’t use bluetooth a lot and haven’t included any of those packages, but there are many of them such as bluez in case you need one.

References:

Linux Desktop Survey Project - Feb/2016

2016-02-07T00:00:00+00:00

I will start this post by thanking the kind folks at the Reddit’s linux subreddit and Voat’s linux subverse who wholeheartedly participated in this survey. Here are the results and they are pretty insightful. For a developer who builds apps targetting the linux platform, it helps to know what are the preferences of users who run those apps on a desktop. Not understanding these requirements beforehand has caused a lot of unneeded backlash in the history of linux desktop development and a lot many forks were created which ended up causing a division of focus.

Not only did I learn a lot about what the linux desktop users want in general, but the survey also threw some light on what distros or desktop environments are generally perceived to be either giving a better performance or providing a better user experience.

What is the most important factor when choosing a Distro?

These were the options I had set:

Better Performance and speed
Better User Experience
A Mix of both

The prevailing assumption of a lot many people is that linux users are usually those “super nerdy” tech geeks who are only interested in arcane things like computing performance and speed. But on the contrary, we learn that linux users are just like Windows or OSX users for that matter, and they prefer a better user experience over performance!

As you can see, most of the folks preferred a balance of both performance and user experience. In fact, Better Performance and speed was the least favored factor for most people who chose their distros, which implies that how wrong our assumptions sometimes could be! As one redditor said, I should have included Software Freedom as an option since that is also quite a motivating factor for a lot many linux users.

Here are two comments from participants who chose the middle-path, A Mix of both:

I don’t feel I have to compromise on one or the other. A good user experience also encompasses acceptable performance.

A good UX is crucial for my own comfort, but the speed and performance must be reliable.

Here is what the undecided folks (Other) have to say:

While user experience is very important, I feel this is for nought if it is not matched with upstream packages. I find it terribly tedious to find myself on a distro which is lagging behind on the likes of gnome-shell and firefox.

Full choice over programs and user interface; my own OS

Stability and security are the most important factors for me.

Another option I should have added here was Stability and security since these are the very foundations upon which linux sits.

People’s Favorite Distro

Ubuntu’s popularity is pretty well-known in the linux world, but what surprised me was Arch. Perhaps, it was the bias effect of lot many people on linux subbredit being Arch fans, or maybe Arch users have actually increased relative to Ubuntu during recent years, who knows! In any case, this survey result does indicate to me that if tomorrow I were to develop a linux destop app, then which distros should I test on with some priority. This is just one of the benefits of conducting this survey.

Here are a few user comments in this category:

Actually Arch derivative Antergos. This distribution has good a user experience and it is performant. It also delivers fresh software.

I use Debian on my desktop, Arch as a DOM0 on my server, and RHEL/CENTOS at work.

I honestly hate the linux desktop. It has so many bugs and is visually very hard to parse.

The Distro with Best Performance

Unlike the Phronix arcane performance benchmarks, I thought about asking the users directly which Distro gave them the best performance. I don’t doubt the benchmarks, but computing is such a complex phenomena that raw performance isn’t as much an indicator of overall performance as it used to be in the past. Besides, the best proof of pudding is in the eating. Linux enthusiasts distro-hop a lot and in that process learn about the distros a lot.

Again, Arch was at top here, followed by Gentoo. Debian came third which was not at all surprising. In fact, I expected this vanilla of a distro to be on the top. Perhaps the “Arch fans” effect of linux subreddit again! What surprised me, however, was Fedora coming such low on performance score. Here is what the participants had to say:

Crunchbang (development started so name changed to Busenlabs)

no idea but xubuntu runs ok on a dual core laptop from 2007 with 2gb of ram.

First benchmarks, then if no existing distro fits the bill, Gentoo, Yocto project, or LFS.

I tested a number of distros on a very slow AMD netbook and Mint with Cinnamon was faster than itself and others with XFCE and MATE.

If performance was the only criteria, I would use windows.

Probably, arch. Whichever one isn’t in the hands of the NSA.

Which Distro gives the Best User Experience

Ubuntu is most people’s favorite in this area, but Ubuntu’s community clone Linux Mint bagging away the top position is also a fitting thing. Like Ubuntu, Mint is a distro which happens to “just work” according to most users. Though I personally don’t like all the extra amount of non-free codecs and plugins it comes pre-bundled with (such as Flash and Adobe), it is a good distro for existing Windows users to hop on. Here is what participants have to say about choosing the distro which provides the best user experience:

Mint is great out of the box and requires minimal configuration to “just work”.

Enhanced user experience means more convenience and more toys; Ubuntu repos cover both.

Arch derivative Antergos. Just enough training wheels to be easy, still all the goodness of Arch. Why not Ubuntu? Ubuntu is bloated and stale and only has the name of being easiest. While once true, the competition has caught up.

Windows (See the same answer under the “performance” question.)

Mint does a very good job of providing the right apps to control settings, out of the box.

Easy to use (Linux Mint), comes with programs that I’m familiar with (Firefox, Thunderbird, LibreOffice) and the interface is simple but also powerful.

Allows heavy customization to how I want it (Gentoo).

Purely for ease of use/install (Ubuntu).

Haven’t tried that many but of of the few that I have tried Mint has had the most clear and logical meny items.

Along with the many enthusiastic distro-hoppers, my search for the discovery of best linux desktop continues.

Eclipse Mars - Not ready for Linux Yet

2015-12-30T00:00:00+00:00

So after hearing about all the hype and praises about this Eclipse new release called Mars, I decided to give it a try on my Ubuntu laptop yesterday. Since I already use Kepler the older version, I was looking for some big positives like performance improvements (most often talked about by a lot of Eclipse fans lately).

When I started this new eclipse on ubuntu, I first had a faint hope that it was running faster (maybe a trick of the mind instilled by the new red logo!). But the true test of any software is how it performs under REAL world conditions. Alas! As I had expected, I was only to be disappointed on the performance front. Unless there is a drastic change in underlying core components such as a code refactoring or an improvement of graphic toolkit/library, the “performance” can only get worse, not better. After I created a simple HelloADT project, here is what happened when I clicked on an Android Activity layout screen:

And this is a modern Intel core i3 machine we are talking about with 4 GB RAM, not some old device. Just after this disaster of an IDE happened, I started my good old kepler version and opened the same ADT project which ran without any problems:

It was good that I had kept my kepler installation folder intact, so I was able to revert. So moral of the story is:

Take all claims about this “big improvement” with a pinch of salt, especially if it is a Java based software.
Wait for Eclipse mars to get more stable before using it for production work.

How to start Android App Development

2015-11-26T00:00:00+00:00

One of the most frequently asked questions on almost any social networking sites these days is how do I start android app programming? Indeed, it is as if Android Development has become some abstract and obscure layer of programming model that only few Illuminati sort of individuals seem accomplished at.

There is no doubt that the process of compiling an android app from source-code to the actual bytecode that runs on your smartphone is a bit more complex than compiling your normal C or Java program. And it seems even more complex than it deserves to be because of some drastic changes Google has introduced with Android Platform, such as:

Android is a whole new Operating System which comes with its own concepts.
Android has a completely new SDK which has both similarities and differences with the Standard Java SE.
Android introduces new programming models for smartphone interfaces like Activity, Intent, etc.
Android introduces a completely new way of building GUI interfaces using an XML dialect.
Android makes frequent changes to its IDE and keeps switching its recommended IDE.

The Android world is always in a flux, though it has been stabilizing a bit since the last year. Eclipse plus the ADT Plugin was the original recommended IDE and still used by many, but a few years back that changed and now Android’s official development way is the Android Studio which is nothing but a decorated version of Intellij IDEA.

In any case, it is important to note that learning Android Development is a full-time job, at least until you grasp the basics. Think of it as just another branch of development, just like you have the traditional Desktop Programming using Visual Studio or .NET, Web Programming using PHP/Python/HTML/CSS, this is the arena of Mobile Apps Development or specifically, Android Development.

But that doesn’t mean you need to know the intricacies of how telephony works or how does your app interact with SIM-cards or the Contacts on your SDCard. Thankfully, Android has already done that low level job for you. Android is based on Linux kernel which is very powerful and secure when it comes to controlling a handheld device. On top of Linux, Android ships with its own libraries that interact with the kernel, along with the Dalvik runtime. Dalvik is nothing but a miniaturized version of your Java Runtime Engine (affectionately called JRE). The thing about Dalvik is that it is much less resource intensive compared to JRE since it is specifically designed for smart devices. Here is the entire android stack or architecture that you should keep referring to througout your app development process:

^Courtesy:\ “Android-System-Architecture”\ by\ Smieh\ -\ Anatomy\ Physiology\ of\ an\ Android.^

The top layer of the stack consists of Applications or apps that we generally use like Contacts, Browser, Home (the Launcher screen that first comes when you start your device), Phone (the dialler) are all apps.

The second layer consists of Frameworks - i.e stuff that the apps are made of. As you will soon learn, there are things like Activities, Views, Content Providers, etc. which are going to be the building blocks of our apps. The Android OS manages these blocks as soon as the user starts our app and starts using it.

Pre-requisites

There are a few things that you already must know in order to start android-programming, the most important being Core Java, also known as Java SE for Standard Edition (not EE or Enterprise Edition). If you don’t know it, then go back to school (or a self-learning website) and learn that first. Java programming introduces enough nuances and complexities of its own, so you will likely get overwhelmed by both Java and Android if you try to learn them at the same time. Recommended Java tutorial site is the one found at Oracle website.

The other thing you should know about is how to use Eclipse or IntelliJ IDEs as you will have to use either one of them to start Android Programming. However, this isn’t a hard requirement like that of Java. IDE is something that you can get used to after using it for a while. It might take some google-searches or errands at StackOverflow at first, but once you get used to it for sometime, you can figure out most things pretty easily.

Thirdly, it is not mandatory but helpful to know some good design principles. The way you style your widgets like TextViews, Labels, EditTexts, etc. very much determines whether people will like your app or not. There is also the thing that when you launch you app in the market, people will judge your app by its cover or front-screen and your download count will likely depend on that. Of course, you can hire a graphic designer for this or if you are a DIY guy like me, learn a professional Image Editing software like GIMP (on Linux) or Adobe Photoshop (on Windows).

Install the Java JDK

Now that you have had your breakfast and kept aside all your distractions to focus on the Android Mission, the first thing to do is install the Java JDK software which is required in order to write and compile Android Apps. Again, head over to Oracle website where you can find the latest download for Java SE. Choose your OS and preferred version. Don’t install Netbeans unless you need it for any other purpose, we will be using either Eclipse or Android Studio for app development.

Install the Android SDK, SDK Tools and Platform Tools

Head over to Android Developer site and download the Android SDK. Now, if you want to use Android Studio (or you are a big fan of the grade build system which is default there), download that IDE which includes the entire bundle. Otherwise, head downwards to find an option called “SDK only” to download the minimal commandline version or if you want to use Eclipse with ADT plugin. The thing here is that you need at least one IDE to develop otherwise it gets pretty complex to hand-code the xml layouts, double-compile your code, first to Java bytecode (.class) and then to Dalvik bytecode (.dex) and finally zipping them into APK (the JAR equivalent), that you are better off delegating these low details to an IDE.

Again, its important to know, why double compilation is required to build Android apps. You see, your code is first compiled to Java bytecode and then, Dalvik compiler takes that output and compiles it to dex or Dalvik bytecode (why that happens is an interesting topic for my another blog post!). Here is how this happens:

Build that Hello World App

Head over to the Configuration Section first to ensure that you’ve setup your IDE properly. Then, the Workflow section is your Bible (or Geeta or Qoran or whatever thrills you). Read it, re-read it, bookmark it, keep referring to it until you are proficient in building and running apps. Needless to say, if you face any issues, there is always Google and StackOverflow which should answer all your queries.

In my next post, I will be covering about android features and a doing some basic things like layout design, interaction with sqlite databases, etc. All the best!

References:

Android Workflow section
Android Configuration Section
Android SDK Downloads
The Android Operating System
Latest download for Java SE

An interesting tale from history: what the Noscript vs. Adblock Plus incident taught us about add-on security

2015-11-09T00:00:00+00:00

Firefox has long been a pillar of the open-source software community, renowned for its commitment to privacy, freedom, and transparency. With strict guidelines for what enters its add-on repository, it’s easy to assume that developers of highly popular extensions share these same values. But sometimes, reality challenges our assumptions in unexpected ways.

Let’s revisit an intriguing tale from 2009 that changed the way we think about browser add-on security: the infamous clash between two popular Firefox add-ons, Adblock Plus (ABP) and Noscript. This story isn’t just a moment of open-source drama—it’s a cautionary tale about trust, transparency, and how we evaluate the tools we rely on every day.

The incident: Noscript vs. Adblock Plus – a code war unfolds

Back in 2009, Noscript, a popular security add-on developed by Giorgio Maone, was partially funded by advertisements. To earn revenue, Giorgio’s add-on quietly whitelisted his own ad-supported websites within Adblock Plus (ABP) without user consent. But it wasn’t a simple whitelist addition; Noscript altered ABP’s internal settings to achieve this, effectively hacking into a fellow add-on’s functionality. This was more than a simple slip-up; it was a violation of user trust.

Naturally, the move didn’t go unnoticed. Wladimir Palant, the developer behind Adblock Plus, discovered the modifications, sparking a public outcry and addressing it in a detailed blog post titled “Attention Noscript users” . What ensued was a very public “code war” between the developers, highlighting a significant vulnerability in how Firefox extensions interacted with each other at that time.

Why this incident still matters today

Noscript’s actions might seem minor in hindsight—especially after Giorgio’s public apology on his blog, “Dear Adblock Plus and Noscript users, dear Mozilla community” —but the implications ran deeper. This wasn’t just about one developer’s questionable judgment. It was a moment that revealed a larger issue: Firefox’s add-on architecture didn’t provide users with adequate control or visibility over permissions.

It also raised broader questions: If a security-focused extension like Noscript could manipulate another extension, what about the myriad lesser-known plugins users were installing without a second thought?

Lessons learned: modern add-on security and transparency

Over the years, browsers have evolved, and so have the security measures around add-ons. Today, permission-based security models are the industry standard, with browsers like Chrome and Firefox explicitly informing users about what an extension can access. Mozilla, in particular, has introduced WebExtensions, a new architecture that offers enhanced isolation between add-ons to prevent similar incidents.

However, it’s not just about architectural changes. This story underscores the need for vigilance when using third-party extensions. Here are a few best practices to keep in mind:

Review Permissions: Always check the permissions requested by an add-on. If it seems to require more access than its function justifies, it’s worth reconsidering.
Stick to Trusted Developers: While popularity doesn’t guarantee trustworthiness, sticking to well-known and highly rated add-ons with transparent histories is usually safer.
Stay Updated: Browser policies and extension capabilities change frequently. Regularly review your installed extensions to ensure they’re up-to-date and still secure.

How far we’ve come: the evolution of add-on policies

It’s interesting to reflect on how far we’ve come since the Noscript-ABP incident. Mozilla’s addons.mozilla.org (AMO) now subjects all add-ons to automated and manual reviews. There are stricter guidelines, better transparency, and a comprehensive permission system in place.

The Firefox add-on saga is a valuable reminder of how even open-source ecosystems are susceptible to lapses in transparency and accountability. It’s a testament to the need for constant evolution in how browsers handle user privacy and security.

A final thought

In hindsight, the Noscript and ABP conflict was more than just a developer feud—it was a pivotal moment that shaped the future of browser security. As users, we benefit from this history, with safer add-ons and better guidelines. But as technology evolves, so do the risks, and it’s on all of us to stay informed and vigilant.

So, the next time you click “Add to Firefox,” remember: every piece of software has a story, and it’s worth taking the time to understand it.

How to add HTML tag highlighting to Geany

2015-11-07T00:00:00+00:00

Ever since I bid farewell to Windows after they introduced that horrible Metro interface in Windows 8, I had been on the lookout for FOSS Linux alternatives that can run easily on my ubuntu 14.04 machine. One such lookout was a replacement for Notepad++ as I always need a handy editor for scripting and other miscellaneous tasks (such as writing this draft in markdown syntax). For large projects, there are always “heavy” things like eclipse and netbeans but I generally tend to avoid them if I can.

Now, the bread-and-butter Linux alternative for Notepad++ is Geany, a light-weight editor developed by Enrico Tröger. Geany has all the wonderful features that Notepad++ has, except one: There is no highlighting available for matching html tag pairs. And as any Web Developer worth his salt will know, an HTML file will soon start turning out to be a huge cosmic soup of tags and braces once the app starts evolving, and this feature can become handy in the situation. And whilst there is a plugin called Pair Tag Highlighter, it isn’t readily available in the ubuntu 14.04 repository. Like many users, I decided to ignore this small nagging inconvenience and moved on. But recently, I was able to spare some time from my daily errands and decided to have a go at it.

The solution to this isn’t that apparent as even my googlefu had to go through a lot of filters before I came across this only post that gave me the solution, though it was a bit more involved and required me to checkout a github repo and build the plugin. This means one of two things: Either a lot of people are not interested in Geany any more and moved on to other editors like Sublime Text, or that they have decided to live with this inconvenience. If you are from the latter group, your inconvenience is soon going to be over. All you have to do is this:

sudo apt-get install git
cd /tmp
git clone https://github.com/geany/geany-plugins.git
cd geany-plugins/
sed -i 's/1.25/1.23/' wscript
./waf configure --enable-plugins=pairtaghighlighter
make all
sudo make install

Things to note:

The fifth line uses the sed tool to change the minimum required version in wscript from 1.25 to 1.23. However, after I’ve written this article, the plugin developer might have raised the minimum requirement and changed it to 1.26 or something. So, replace accordingly.
Make sure you have the latest version of python installed in order to use the waf tool. You also need to install intltool package if it does not already exist.
If all goes well, restart Geany and go to Tools->Plugin Manager and enable the Plugin Pair Tag Highlighter. Once you do that, your HTML code will start highlighting the opening and closing tags as shown below:

References:

Sqlalchemy Hack - How to convert a table to dict on the fly

2015-07-04T00:00:00+00:00

In on of my recent projects, I came across the need to develop a JSON based REST API to fetch data from the sqlalchemy objects. Now, the Query object is a great way to access data using the powerful sqlalchemy orm, but it doesn’t give any built-in way to convert the result-set into a python dict.

For instance, I have a Professor table in my models.py. Here is what I get when I query the Professor model using sqlalchemy ORM:

>>> session.query(models.Professor).all()
2015-07-05 11:07:57,282 INFO sqlalchemy.engine.base.Engine SELECT professors.id AS professors_id, professors.name AS professors_name,
partment, professors.email AS professors_email, professors.password AS professors_password, professors.phone AS professors_phone
FROM professors
2015-07-05 11:07:57,282 INFO sqlalchemy.engine.base.Engine ()
[<Professor(name=`Albus Dumbledore`)>, <Professor(name=`Severus Snape`)>]

At the end, I got the two Professors’ name in a collection, but they are still sqlalchemy objects. How can I convert it to a python dict, so that it can be deserialized to JSON or be used for any other purpose. To solve this, just add the below code to your models.py module, and just call the models.to_dict() method to convert an sqlalchemy models collection to a dict (or alternatively, call models.from_dict() to build the model object collection from an existing dict!):

def to_dict(model_instance, query_instance=None):
	if hasattr(model_instance, '__table__'):
		return {c.name: str(getattr(model_instance, c.name)) for c in model_instance.__table__.columns}
	else:
		cols = query_instance.column_descriptions
		return { cols[i]['name'] : model_instance[i]  for i in range(len(cols)) }

def from_dict(dict, model_instance):
	for c in model_instance.__table__.columns:
		setattr(model_instance, c.name, dict[c.name])

The usage is as follows:

q = dbsession.query(models.Application).filter(models.Application.id==id)
professors = q.all()
di = models.to_dict(professors) #for converting a single table resultset

q = dbsession.query(models.Application)
application = q.first()
dd = models.to_dict(app)
dd['student'] = models.to_dict(application.student, q) #for converting a relationship object which refers to another table. 

The reason we have to use a slightly different approach for the relationship objects (like application.student which refers to the student model) is that the instance doesn’t have an __table__ object which is required to build the dict. Hence, the query instance is passed. This is just one approach I’ve used to convert sqlalchemy objects to dict. If you have any other approach, please let me know.

Is Sourceforge.net safe anymore to download software

2015-06-25T00:00:00+00:00

You have probably heard about all the recent buzz surrounding the sourceforge.net website hosting adware/malware bundled installers and naive users falling prey to it. In some cases, SF even actively took control of abandoned developer accounts. to bundle their crapware.

So, is it safe anymore to go to this site and download your favorite FLOSS software? The TLDR version is NO, unless you happen to trust the developer and the project is actively being developed by them. But to understand the big picture, you will have to see the root cause of this which goes much deeper than just sourceforge, and relates to software distribution practices in the Microsoft Windows world.

What went wrong?

Software Development consists of two important phases: Authoring of source code, and secondly, the build process where the code is compiled to binaries and bundled into installable packages that you may download. Many years ago, sourceforge was one of the rare sites that provided a place on the internet where open source developers can host their code and binaries, and users can download them. This used to go well for many years and thus sourceforge gained a nice reputation in the FLOSS Community.

But soon enough, Sourceforge decided to cash in on this reputation by breaching this trust. And they started bundling adware and crapware to the developer builds. In some cases, the lead developers were in cohorts with Sourceforge. And as this Windows Club article succintly explains, Sourceforge wasn’t alone in this. There were many othere like CNET.com, Download.com, etc.

The Problem

Though the adware bundling is optional in theory, the default options are so tactfully placed that only the shrewdest of hackers can dodge all of them! Though this kind of behavior is sort of expected from B-grade sites like CNET and download.com, Sourceforge was something that the open source community held in high esteem. Though most of the developers have moved to github now, souceforge was once considered the Jerusalem for downloading FLOSS Software, especially Windows Software. Even today, their interface is the best suited for novice users who are just looking for a big green download button to click - Github interface is more geared towards developers and generally overwhelms them. But now that the Jerusalem folks have turned into heretics (for lack of a better analogy), what are our options?

The best option is to get it from the developer’s site itself if they provide it. But sometimes that’s not possible as the developer may not always have bandwidth to host the binaries, and we have to depend on 3rd party sites like sourceforge. As the author of the linked article suggests, MajorGeeks and Softpedia are generally clean in my experience. Downloading software from those cleaner sites, however, is just a workaround and doesn’t solve the larger structural problem of Software Distribution in Windows world. So, what is that solution?

The Linux Solution!

One crisis we are facing in the Windows FLOSS world right now is the Middleman problem. The original developer (A) writes some code and compiles it, the middleman (SF) bundles adware to it and makes a quick buck, whilst an innocent user (B) clicks that download link. Since there is no way for B to verify if the package is actually bundled by A, SF continues to get his commissions.

But what if there was a way for B to verify if the package is actually signed by A? In the Linux world, we do have such a mechanism, so there is no place for middlemen to make a profit there. If I download GNU Emacs from the GNU ftp server, all I have to do is gpg --verify emacs.sig emacs.tar.gz and the system will tell me whether my emacs copy is signed by developers of Emacs, or was it tampered with for injecting any malware!

The second crisis the Windows FLOSS is facing is lack of a decent package management system. In above analogy, we can’t expect B to personally verify each downloaded software against A’s signature, s/he needs a good package manager that downloads the package from a trusted server and installs it afer verifying the signatures.

In Linux world, we have two major packaging systems known by the name of apt and yum. How are they better, stable and secure than the Windows approach of users downloading arbitrary installers from SF-like sites and installing them, is something beyond the scope of this article. But suffice it to say that its very very high time that Windows gets a similar package manager to take care of the present structural issues surrounding it. Ninite has tried to do something in this direction, but its a proprietary and paid solution. We need open source and something that everyone accepts as a standard. Maybe, perhaps Microsoft will take the initiative and do it themselves? Only time will tell.

How to know whether my blog post will attract traffic or not

2015-06-10T00:00:00+00:00

One of the evergreen topics for discussions in the blogging world is SEO, and especially the various ways of attracting bless of the all-pervading search engines to one’s own site or blog. There are various ways to approach this problem, some of the popular ones are:

Using meta tags like keyword and description to attract the search engines.
Performing keyword research (using Google’s planner tool or similar).
Getting link juice through social media marketing i.e creating backlinks.
Hiring SEO experts or web designers to get some more arcane suggestions.

But lost in these technical details, folks are forgetting one basic foundation upon which the Internet is built: Content is King. Yes, the sad truth is that Google (or Yahoo/Bing for that matter) cannot be fooled by any of these tools in the long term. Doing some social media marketing on Facebook might place your article on #1 position for a short period of time, but sooner or later, Google will figure out whether your content is actually worth it or not.

But how do you write good content, and also ensure that it has some uniqueness that the million other sites on the Internet don’t already have? Well, I can’t help you with the first question. Since only you know what your area of expertise is, only you can come up with a good topic and subject-matter for your blog or site. For instance, Web Development is my area of expertise, and I came up with this particular topic you are reading right now.

But I can help you with the second question. Doing some research on this matter and performing a topic-wise traffic analysis of my blog has proven this one thing: Content is King. So, my method of figuring out whether an article/post will rock or not, is based upon this simple canon. Once you decide the topic and subject-matter of your next blog post, do this simple reality check before you even bother drafting it:

Almost every googler is looking for a specific solution for a problem or issue, be it shopping for a product, knowing a URL address or researching something. So, your next post or article should solve that problem in order to pass the content test. To get a quick idea of the competition, perform a google search from the perspective of your potential googler! A glance at the results will instantly tell you whether this problem is already solved, and by how many.

For instance, before publishing this article, I did this simple Google search:

How to know whether my blog post will attract traffic or not

And this is what I got:

All those are top ranking sites and deserve to be on page one (no offense meant to them). However, for someone specifically looking for a solution to this question (agreed, however small that set of users be), none of them have an immediate answer that could be surmised by the search result summary. Whilst they are all making clear points about ways to increase search engine traffic, none of them talk about knowing, in priory, whether a given method will work or not. So, at least for this particular query (maybe used by about 1% of all bloggers), this blog post deserves to be amongst the page one results.

How to host a Flask app on Openshift

2015-02-06T00:00:00+00:00

Note: This article may no longer be relevant as Red Hat has recently changed the openshift stack with docker and kubernetes.

Openshift free tier is an excellent way to host your python web app for staging or testing, and you can even host a low to medium traffic production site. Openshift provides several options (cartridges) for hosting including python, php, node.js, etc. but python being my favorite language and Flask being a minimalist and flexible framework, this combination is what I usually end up with.

Create an Openshift account

In case you haven’t already, head over to Openshift and sign up for a free tier. You will be able to host at most three apps for each account. Openshift apps are hosted on rhcloud.com domain and you’ll have to setup a subdomain first which will be part of your app url. For instance, if I register prahladyeri.rhcloud.com subdomain, I can create the following apps with that:

myflaskapp-prahladyeri.rhcloud.com
myphpapp-prahladyeri.rhcloud.com
blog-prahladyeri.rhcloud.com

Install the necessary tools

You will need the following:

Git: Your app resides in a git repository, so you’ll need git installed on your machine to push changes. App is deployed automatically once you push.
Python and Flask: Obviously, you are going to need them if you are building a Flask app.
Openshift rhc tool: This ruby based tool is optional, only use it if you don’t want to use their online portal for creating apps or you aren’t familiar with ssh. Personally, I didn’t want to install ruby on my machine just for this one purpose.

Pull the remote repo

Once you create a python app, Openshift will provide you a git repository url as follows:

Now open your command line and pull this starter repo to your local machine:

git clone <YOUR_SOURCE_URL> myFlaskApp

Now, the remote repository will be cloned in the myFlaskApp folder. Browse it to see the scaffolding.

Add your flask app

The scaffolding structure will be as follows:

wsgi.py
setup.py
.openshift/..
.settings/..
wsgi/..         => your python source files go here.
wsgi/static..   => your static folders viz css, img, fonts, et al. go here.

If the wsgi/ folder doesn’t exist, you’ll have to create it. Just modify the setup.py and add Flask and SQLAlchemy as your app dependencies along with your app name. This tells openshift to make sure that dependency packages are available whenever you push any code changes.

from setuptools import setup

setup(name='myFlaskApp',
	  version='1.0',
	  description='myFlaskApp',
	  author='Prahlad Yeri',
	  author_email='prahladyeri@yahoo.com',
	  url='http://www.python.org/sigs/distutils-sig/',
	  install_requires=['Flask==0.10.1','SQLAlchemy==0.9.8'],
	 )

Now, create a text file named application in the wsgi/ folder with the following contents:

#!/usr/bin/python
import os

virtenv = os.environ['OPENSHIFT_PYTHON_DIR'] + '/virtenv/'
os.environ['PYTHON_EGG_CACHE'] = os.path.join(virtenv, 'lib/python2.7/site-packages')
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
	execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
	pass

from myFlaskApp import app as application       

This is a configuration file that tells openshift where your Flask app script resides. Now create a python file called myFlaskApp.py, this will be your HelloWorld script:

import flask
from flask import Flask
from flask import request

app = Flask(__name__)

@app.route("/")
def home():
	return "Hello World"

if __name__ == "__main__":
	app.run(debug=True)

The last part of the code (app.run) is there so that you may test the Flask app by running this script on your local machine before pushing these changes.

Push your changes

All that is left to be done now is committing your changes and pushing them to openshift:

git add .
git commit -m "Initial commit for myFlaskApp"
git push origin master

Voila! You are done

Wasn’t it almost as easy as deploying a php script on your web host? If everything goes right, your Flask app will be hosted on http://myFlaskApp-mydomain.rhcloud.com/. Visit your app link and check it out.

Few important things

In case you aren’t familiar about how git over ssl works, the remote machine authenticates your machine using an SSL public key you have already provided them. So if this is your fist time, you’ll have to generate a private-public key pair (using ssh in linux or putty on windows). After that, you’ll have to update your public key to Openshift, so they can authenticate your machine. You can add it using the Settings menu on the Openshift portal.
Its very important that all your static files reside in wsgi/static folder and that folder only. Openshift uses that path by default. But in case you are really stuck with using /css and /js in your existing app, as a solution you can clear the static_url_path in your flask app as follows:
```
app = Flask(__name__, static_url_path='', static_folder='static')
```

Refer to this Openshift tutorial for more details.

Make the most of SSH. Some times, you may want to connect with the remote server using secured shell (ssh/putty) for troubleshooting, viewing logs, etc. Your SSH url is included in your git source url. So, if your git url is of the form:
```
ssh://500XXXXXXXXXXXX01061a@prahladyeri-inn.rhcloud.com/~/git/prahladyeri.git/
```

Just remove the ssh:// from the beginning and the other things after the domain, so the SSH host url becomes:

500XXXXXXXXXXXX01061a@prahladyeri-inn.rhcloud.com

References:

Migrating from WordPress to Jekyll: save money with a static site

2015-02-04T00:00:00+00:00

Here I am, signing off from a self-hosted WordPress site and finding a welcome change in Jekyll, a blog-aware static site generator. There is nothing new about this, several well-known bloggers have already migrated to Jekyll in the last few years. Ever since Tom Preston Werner created this software in 2008 and published his infamous article about Blogging Like a Hacker, it has become the go-to thing for at least the small and indie bloggers.

While WordPress is a powerful platform, it can feel over-engineered for simple blogging needs. For indie bloggers who don’t need the complexities of a RDBMS like MySQL, a static site could be a more efficient solution. The hassles of administering and maintaining various themes and plugins could feel overwhelming at times. However, since there weren’t too many alternatives in those days and PHP hosting was an easy path, many ignored this factor.

How Jekyll can save you hosting costs

Jekyll generates static sites made of pure HTML/CSS. Static sites, unlike dynamic ones, don’t require server-side processing or database queries, which reduces hosting resource usage and speeds up load times. This makes static sites both cost-efficient and faster. In fact, Github pages provides you fully free static hosting for Zero USD per month!

Jekyll vs WordPress: A closer look

While both Jekyll and WordPress serve the purpose of creating websites, they differ in several key areas, making Jekyll a better choice for specific use cases like simple blogs or static sites.

Speed and Performance: WordPress relies heavily on server-side PHP scripts and database queries to dynamically generate pages, which can slow down site performance, especially if not optimized. On the other hand, Jekyll pre-generates static HTML pages, significantly reducing load times as there’s no need to process requests or queries. Static sites also tend to perform better under heavy traffic since they’re served directly from the server without any backend processing.
Security: WordPress’s widespread usage makes it a common target for hackers, particularly due to vulnerabilities in plugins, themes, or unpatched core software. Jekyll, by contrast, is far less prone to security breaches because static sites don’t require a database or server-side processing, reducing potential attack vectors. With Jekyll, you don’t have to worry about plugin updates or securing a backend.
Customization and Plugins: WordPress shines when it comes to flexibility through its vast ecosystem of plugins and themes. However, this can lead to bloat, slowing down your site. Jekyll’s simplicity means fewer customization options compared to WordPress, but it also means less overhead. Instead of relying on plugins, you can customize your Jekyll site directly through code, giving you full control without unnecessary baggage.
Maintenance: WordPress sites require regular maintenance, such as updating plugins, themes, and the WordPress core itself. This can be time-consuming and may lead to incompatibilities. With Jekyll, maintenance is minimal—once your site is deployed, it’s mostly hands-off. There’s no need to manage databases, perform software updates, or worry about downtime due to version conflicts.

In essence, if you’re running a content-heavy blog that doesn’t need dynamic features or heavy customization, Jekyll can save you time and resources. For more feature-rich or complex sites, WordPress remains a powerful choice but comes with its own set of maintenance responsibilities.

SEO and other challenges with Jekyll

While Jekyll offers numerous benefits in terms of speed, cost, and simplicity, there are a few considerations that you should keep in mind before making the switch.

SEO (Search Engine Optimization): In WordPress, SEO optimization is often handled through plugins like Yoast, which makes it easy to tweak meta tags, sitemaps, and other SEO-related elements. With Jekyll, these features aren’t built-in, and you’ll need to configure your SEO manually. This means writing metadata directly into your HTML or Markdown files and creating your own XML sitemaps. Fortunately, there are Jekyll plugins for generating sitemaps, RSS feeds, and optimizing metadata for SEO, but setting them up might require some coding knowledge.
Dynamic Content: One of WordPress’s biggest advantages is its ability to handle dynamic content like comments, forms, or membership systems. While services like Disqus or Staticman can add dynamic features like comments to Jekyll sites, they don’t offer the same level of functionality or flexibility that WordPress does. For bloggers who want more interactive features like user logins, contact forms, or complex ecommerce capabilities, WordPress may still be the better option.
Learning Curve: While Jekyll is simpler in terms of maintenance, there’s a learning curve when it comes to getting started. You’ll need to be comfortable with the command line, Git, and Markdown, as well as basic HTML and CSS for customizations. While WordPress allows users to manage their site through a graphical interface, Jekyll requires a more hands-on approach, which might be daunting for beginners.
Third-Party Integrations: WordPress integrates seamlessly with various third-party services through its plugin ecosystem, from payment gateways to email marketing platforms. Jekyll, while offering more control, may require additional effort to integrate with these services manually.

Despite these challenges, Jekyll’s advantages—especially in terms of speed, security, and cost—far outweigh the downsides for users looking for a simple, fast, and secure platform for static content.

The implementation

As any seasoned PHP programmer would tell you, programming in a language like PHP isn’t everyone’s cup of tea. While you don’t have to do any programming for hosting a static site, you should still know about a few things in order to implement a static site:

Jekyll: Jekyll is a static site generation tool written in Ruby language to generate blog-aware static sites (like the one you are presently reading). Visit the link to find documentation that explain what Jekyll is, how to install Ruby and Jekyll on your system and publish your posts with it.
Markdown: Markdown is a utilitarian formatting language crafted specially with online publishers in mind. Jekyll posts written in Markdown syntax are saved with the extension of *.md. While you can write posts in HTML syntax too, knowing markdown comes very handy and saves time.

For instance, when I write:

[Jekyll](http://jekyllrb.com/)

It becomes:

Jekyll

Making lists, headings, etc. is as easy. For instance,

### Three hashes means H3
#### Four hashes means H4
1. This is list-item1
2. This is list-item2

becomes:

Three hashes means H3

Four hashes means H4

This is list-item1
This is list-item2

Tools of the trade

Disqus: Perhaps the only feature for which most indie bloggers need a dynamic site is that of posting comments. Thanks to services like Disqus, bloggers can now leave the hassle of maintaining their own database for storing their readers’ comments. Disqus does this for them.
Github pages: GitHub Pages is an excellent option for bloggers looking for a reliable, free static web hosting service. It offers generous bandwidth and content hosting, making it a great choice for many users, especially those who want to keep costs low while maintaining flexibility.
Google Analytics: Google Analytics integrates seamlessly with Jekyll, providing valuable insights into your audience demographics and behaviors without needing additional WordPress plugins.

Outcome: Why Jekyll is worth the switch

After making the switch from WordPress to Jekyll, I can confidently say that the benefits of a static site far outweigh the initial setup effort. From improved site performance and lower hosting costs to greater security and minimal maintenance, Jekyll has proven to be a highly efficient platform for my blogging needs.

If you’re running a simple blog, personal portfolio, or documentation site, and you’re tired of managing a database and paying for expensive hosting, Jekyll might just be the solution you’ve been looking for. By leveraging GitHub Pages and static site generation, you can enjoy the peace of mind that comes with a fast, secure, and cost-effective website.

Take the leap today and explore Jekyll as your next blogging platform. With plenty of resources and guides available, including some I’ve listed in the references and migration checklist below, you’ll find the transition smoother than expected.

Migration checklist

Step	Description	Tools/Resources
1. Backup Your WordPress Site	Before making any changes, back up your WordPress site, including the database and files.	UpdraftPlus, All-in-One WP Migration
2. Install Ruby and Jekyll	Set up Ruby and Jekyll on your local machine to create and manage your static site.	Jekyll Installation Guide
3. Export WordPress Content	Use a plugin or WordPress’s export tool to export your posts, pages, and media.	Jekyll Exporter Plugin, WordPress Export Tool
4. Convert WordPress Content to Jekyll	Use the exported content to convert it to Jekyll’s format, which involves generating Markdown files.	Jekyll Exporter Plugin
5. Set Up Your Jekyll Site	Create a new Jekyll site and configure your theme, layouts, and plugins.	Jekyll Docs, Jekyll Themes
6. Customize Your Jekyll Site	Modify the theme, layout, and styles to match your old WordPress site or give it a fresh design.	HTML/CSS, Poole Jekyll Theme
7. Add Comments via Disqus	Use Disqus to handle comments since Jekyll doesn’t support dynamic comment systems.	Disqus for Jekyll
8. Implement SEO	Set up SEO by adding meta tags and optimizing content. Use Jekyll plugins for sitemaps and meta tags.	Jekyll SEO Plugin
9. Set Up Analytics	Integrate Google Analytics to track your site’s performance.	Google Analytics
10. Host Your Jekyll Site	Choose a hosting service like GitHub Pages, Netlify, or your own server.	GitHub Pages, Netlify
11. Test Your Jekyll Site	Test your static site to ensure everything is functioning as expected.	Browser, Jekyll Local Server
12. Migrate DNS to New Hosting (Optional)	If you’re using a custom domain, update your DNS settings to point to the new host.	GitHub Pages Custom Domain Setup
13. Finalize Migration	Once tested, finalize the migration by making the Jekyll site live and notifying users of the change.	N/A

Reference:

4 Ways to share your mobile internet with a PC/laptop

2014-12-20T00:00:00+00:00

The one thing I like about the world of Linux in general and also Android is the great number of options. Unlike Windows phone, where there is only one way to perform a task (if at all there is!), android provides a richer user experience in all departments.

Suppose you are traveling to some place and your laptop has lost its usual source of internet connectivity. You will naturally consider surfing the web on your cell-phone, or better still, try and find a way to share the internet on your cell-phone with your laptop, so your work won’t suffer.

The android OS supports multiple ways to perform tethering (a.k.a share your mobile internet with other devices such as laptop).

USB Tethering: I mention this method first because it is the most power-efficient of the lot. Always try to use this method before others if you want to prioritize battery savings. The only shortcoming of this method is that you will only be able to share the mobile internet with at most one device only (in the above example, your laptop). And of course, you will be able to keep your USB cable handy. To use this method, just hook up the USB cable to your laptop (making sure that the device drivers are installed). Then, on your cellphone, go to Settings->Tethering and Portable Hotspot and tap on the USB Tethering option to activate it. (On your particular model, the actual setting name might differ as each manufacturer customizes android to their own liking)
Bluetooth Tethering: This method is less efficient than USB tethering, but still quite efficient compared to others. Again, the disadvantage is that at most only one peer can be shared. Also, the speed will be much slower compared to Wifi and other techniques, but if 10mbps is workable for you, then this option could be a great help for you. To activate, tap on this option in Settings->Tethering and Portable Hotspot.
Wifi Access Point: This method is the most common of the lot as the concept of turning your cellphone into a Wifi access point has turned into quite a fad nowadays. However, it is also the least efficient energy wise. The advantage, of course, is that your cellphone can provide its bandwidth to multiple devices at the same time. To activate, tap on this option in Settings->Tethering and Portable Hotspot.
Using a USB Dongle as Mobile broadband: This method applies only to GSM radios (not CDMA as they are locked by providers). Though it doesn’t come under the definition of tethering, the effect is same. If none of the above methods work for you (or for some reason, you don’t want to use), as an alternative, you can take out the SIM card and put it in an unlocked USB dongle (also called Data Card). Data Cards are manufactured by companies like ZTE and Huawei and unlocked versions are available in many online retail stores.

7 Reasons I would prefer an Android Phone over WP8

2014-12-19T00:00:00+00:00

My existing phone is a Karbonn A30 running Android ICS and my next phone is going to be an Android phone too. Both as a user and as a developer, I’ve come upon the realization that Android provides a much better user experience and overall satisfaction in mobile computing.

I had briefly considered using Windows Phone once, but I’ve discarded that idea after giving it a trial run. Here are the 7 reasons why I would only go for a phone running the Android Operating system and not WP8.

More features: This is one of the primary reasons a user buys any product, let alone a cell-phone. Android clearly wins in this department as it provides more features (out-of-the-box) compared to WP8. Just open the android settings menu and compare it to the feature-deprived WP8 settings screen to prove this to yourself. WP settings doesn’t even allow you to perform basic things like:
1. Setting up a custom mp3 ringtone (You need a PC with Zune software running for that.)
2. Setting up bluetooth options (No advanced options like bluetooth tethering )
3. Brightness settings are only low, medium and high (whereas Android provides a slider to set a custom percentage).
4. No built-in file manager (At least the lumia 520 I tested did not)
5. Not enough power stats (Android shows a complete run-down of what apps ate how much battery life, WP doesn’t even show the exact percent of battery remaining)
More options for tethering: One of the major reasons that I use my phone is to tether my phone’s internet, so as to use it on my laptop. Android provides me three options to do this: Wifi tethering (aka Access point), Bluetooth tethering, USB tethering. WP8 only provides Wifi sharing and that too is not quite customizable.
More free apps: The Google play store is full of professional apps (many of them free, too). For instance, ES3 is a file-manager, archiver, FTP-manager and backup-manager combined. This app is not freely available on WP8 and I doubt the paid version would be having the features comparable to android version. Other examples are Terminal-IDE (A fully-fledged IDE) and QPython (A fully-fledged programming environment!) that are both freely available on Android.
Opensource technologies: Android stack completely rests on Open source technologies. The Linux kernel, Apache Harmony and Dalvik VM are all Opensource components.
Supported by Google: Another reason to go with Android is that it is actively developed and supported by Google Inc, a company whose interests directly harmonize with that of the end-users.
More options for the Developer: Even as a developer, you will have a much better time developing an Android app than a WP8 app. First of all, the development tools (EclipseADT and Android SDK) are pretty well-documented on Android Developer site. Secondly, these are opensource tools that run on all platforms including Linux and OSX. On the other hand, developing a WP8 app requires Windows 8 64 bit version and nothing else.
No vendor lock-in: As a corollary to point 4, there is no vendor lock-in when you go for Android. Because the source code of android OS is freely available from the repository, you can customize it if you need to.

9 Optimizations to make your Linux Desktop fly like a Rocket!

2014-06-28T00:00:00+00:00

This article is the result of notes I’ve prepared during tweaking, twisting and optimizing ubuntu variants over the last few years. In case you use any other distro, some of these settings may not be applicable to you. For best results, these changes must be done on top of a fresh installation, otherwise chances of things breaking increase a bit. Each step is optional - In case of software removals, do it only if you are not going to use the concerned software. Be careful before making any changes and know exactly what and why you are doing that.

#1 Optimize disk access with noatime:

Each file and folder on your linux system has a file-creation timestamp and a modification timestamp. Apart from that, linux tries to keep track of “access time” for each of these files. Now keeping track of the access time has its performace-cost, and if you want to remove this performance cost, you need to specify “noatime” attribute in the disk partition entries in your /etc/fstab file. Edit this file in your text-editor and set noatime as follows:

UUID=97102801-14e3-9752-7412-d9c57e30981w / ext4 errors=remount-ro 0,**noatime** 1

#2 Optimize Swappiness:

Swappiness is the tendency of the linux kernel to prefer disk-swapping vis-a-vis physical memory. The default swappiness value of 60 was kept considering server installations. If you are a desktop user having a machine with good RAM, you would normally prefer disk-swapping to be minimal. You can safely reduce this value to 10. To do so edit the file /etc/sysctl.conf and add the following:

vm.swappiness=10

(Just change the entry if it already exists, don’t make a duplicate!)

#3 Install preload:

If you typically use the same programs regularly, preload will help you by loading into memory, the programs that you use most frequently. To install on ubuntu:

sudo apt-get install preload

#4 Place your mission critical apps in /dev/shm:

Few weeks back, I was having performance issues with running Eclipse on ubuntu. After tweaking and optimizing various JVM settings in vain, the thing that really made the difference was placing the entire JDK folder in RAMDISK. The /dev/shm folder is like a virtual ramdisk (on ubuntu and derivatives) where you can place your temporal, high-priority stuff to run it in “best performance” mode.

#5 Remove unwanted programs from startup:

Many linux distros such as ubuntu come loaded with a ton of baggage, and if you are someone like me, you might feel obliged to reduce some burden off your system by removing or disabling unwanted software and daemons from it. You can do this by going to “Startup Applications” in the System menu, but ubuntu hides the pre-installed apps by default. To overcome this limitation, open your terminal and issue the below command:

sudo sed -i 's/NoDisplay=true/NoDisplay=false/g' /etc/xdg/autostart/*.desktop

{.alignnone .size-full .wp-image-3092 width=”649” height=”612”}

Now you can go through the startup programs list and can disable the unwanted ones. Common sense will tell you that if you don’t use bluetooth on your machine, you can get rid of the “Bluetooth Manager”. Similar is the case with “Backup Monitor” in case you don’t need to sync your backups in realtime. Here is the list of services that I’ve safely disabled without causing any issues:

Backup Monitor
Bluetooth Manager
Chat
Desktop sharing
Gwibber
Ocra screen reader
Personal file sharing
Screen saver
Ubuntu one
update notifier

#6 Uninstall software that you don’t use:

The next step is to remove those software that you don’t use at all. Again, some common sense but with some caution is needed here. There are some programs (like empathy) that form the core part of ubuntu, so it won’t allow you to “apt-get remove..” them without removing unity itself. In such cases, we will disable such programs from starting up as services (next step). Some of the programs that you may safely remove are:

apt-get remove samba-common\
apt-get remove cups\
apt-get remove avahi-daemon avahi-autoipd

I typically uninstall all three after a new installation. The first one is needed for file-sharing in the local network if you have one. Second is the print daemon, and third is used to broadcast common network services across the local network and finding local hosts by using friendly names like “local.workstation”.

#7 Disable unwanted daemons:

In case you don’t want to remove the cups program as you might need printing in future, you can disable them for the time being. To do so, issue the below command:

sudo sh -c "echo 'manual' \>\> /etc/init/cups.override"

You may disable any daemon in this manner by doing a manual override, just replace the “cups.override” with the daemon name that you want removed such as:

sudo sh -c "echo 'manual' \>\> /etc/init/bluetooth.override"

Later, if you want to enable that daemon, all you to do is delete the .override file.

#8 Optimize Nautilus to behave in a speedy manner:

This is totally optional. Nautilus, by default, tries to show thumbnails of each and every file in a directory. If the directory contains a lot of files, this causes a noticable delay. Now if you are in the habit of regularly previewing thumbnails of your images, don’t do this optimization. Otherwise, if previewing thumbnails don’t matter to you and all you are interested in is speed (like me), you can go to Edit->Preferences->Preview-tab and set the preview settings to Never.

#9 Disable translation downloads in aptitude:

This setting is for speeding up the downloads from apt repositories rather than your machine. By default, ubuntu adds additional translation repos when you issue “apt-get update” command to update your repository settings. If you only need english, you can disable translation downloads by editing /etc/apt/apt.conf.d/00aptitude and additing this line to it:

Acquire::Languages "none";

References:

PHP-FPM vs node.js - The REAL Performance Battle

2014-06-22T00:00:00+00:00

Even after my last article about PHP and node.js benchmarking, my search for the holy grail of performance truth still continues. However, I do understand now that pitting PHP running on apache against a stand-alone node was a bit unfair with PHP for it was limited by what the apache configuration could handle.

{.alignnone .size-full .wp-image-3082 width=”1021” height=”622”}

No, this time I went with nginx, a light and performance oriented server that was specifically designed to solve the C10K problem from the ground-up. And who better than PHP-FPM, the enhanced Fastcgi process manager that implements asynchronous features (at least in theory) to take on node.js. node.js is the one server that implements all its features primarily using callbacks in javascript, and thus drastically improvising performance by leveraging the benefits of functional programming (again, in theory).

I used the same code I had used earlier but did a small improvement to it so that the random filenames generated for performing I/O are unique:

<?php 
//asyncdemo.php
$s=""; //generate a random string of 108KB and a random filename
$filename="";
//generate a random filename
do {
	$fname = rand(1,99999).'.txt';
} while(file_exists($fname));

//generate a random string of 108kb
for($i=0;$i<108000;$i++)
{
	$n=rand(0,57)+65;
	$s.=chr($n);
}

//write the string to disk
file_put_contents($fname,$s);

//read the string back from disk
$result = file_get_contents($fname);

//write the string back on the response stream
echo $result;

And here is the Javascript version:

//server.js
var http = require('http');    
var server = http.createServer(handler);
var fs = require('fs');

function handler(request, response) {
	response.writeHead(200, {'Content-Type': 'text/plain'});

	//generate a random filename
	do{fname = (1 + Math.floor(Math.random()*99999999))+'.txt';
	} while(fs.existsSync(fname));

	//generate a random string of 108kb
	var payload="";
	for(i=0;i&lt;108000;i++)
	{
		n=Math.floor(65 + (Math.random()*(122-65)) );
		payload+=String.fromCharCode(n);
	}

	//write the string to disk in async manner
	fs.writeFile(fname, payload, function(err) {
			if (err) console.log(err);

			//read the string back from disk in async manner
			fs.readFile(fname, function (err, data) {
				if (err) console.log(err);
				response.end(data); //write the string back on the response stream
			});  
		}
	);
}

server.listen(8080);
console.log('Running on localhost:8080');

So, what happens when we run a piece of web application code performing async I/O for two thousand times (with two hundred concurrent) using a tool like apache-bench? Who is faster - PHP-FPM or node.js? Here is the answer.

So, moral of the story is that even the latest and greatest of PHP world falls behind node.js (though by a much smaller margin than before). Now, I do understand that PHP’s market is very large, and with so many opensource CMSes like wordpress, mediawiki and drupal already powered by PHP, it is quite difficult to shake PHP’s market share in the near future.

On the other hand, with the performance advantage that node.js offers, its a very lucrative option for startups small businesses that don’t have the funding to develop high-end enterprise apps in say, Java or SAP. More importantly, if tommorrow I were to given a task of developing a performance-driven app, is there one reason why I should not write it in node.js and go for PHP-FPM instead? Some food for thought. Comments are Welcome!

Summarized results:
PHP-FPM: 64.447 secondsnode.js: 42.441 seconds

The Machine:
Intel Pentium Dual-Core 2.30GHz running Linux 3.2.0

The Configurations:
PHP-FPM: PHP 5.4.23 (fpm-fcgi) (built: Jun 22 2014 14:51:15
NODE: node v0.10.28

Detailed Results:
--PHP-FPM-----
ab -c 200 -n 2000 http://localhost:8080/asyncdemo/asyncdemo.php

Concurrency Level:      200
Time taken for tests:   64.447 seconds
Complete requests:      2000
Failed requests:        6
   (Connect: 0, Receive: 0, Length: 6, Exceptions: 0)
Write errors:           0
Non-2xx responses:      6
Total transferred:      215649378 bytes
HTML transferred:       215355222 bytes
Requests per second:    31.03 [#/sec] (mean)
Time per request:       6444.742 [ms] (mean)
Time per request:       32.224 [ms] (mean, across all concurrent requests)
Transfer rate:          3267.70 [Kbytes/sec] receive


--NODE-----
ab -c 200 -n 2000 http://localhost:8080/

Concurrency Level:      200
Time taken for tests:   42.441 seconds
Complete requests:      2000
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Write errors:           0
Total transferred:      216155440 bytes
HTML transferred:       215953440 bytes
Requests per second:    47.12 [#/sec] (mean)
Time per request:       4244.115 [ms] (mean)
Time per request:       21.221 [ms] (mean, across all concurrent requests)
Transfer rate:          4973.69 [Kbytes/sec] received

PHP vs node.js: The REAL statistics

2014-06-09T00:00:00+00:00

When it comes to web programming, I’ve always coded in ASP.NET or the LAMP technologies for most part of my life. Now, the new buzz in the city is node.js. It is a light-weight platform that runs javascript code on server-side and is said to improvise performance by using async I/O.

The theory suggests that synchronous or blocking model of I/O works something like this:

I/O is typically the costliest part of a web transaction. When a request arrives to the apache web server, it passes it to PHP interpreter for scripting any dynamic contents. Now comes the tricky part - If the PHP script wants to read something from the disk/database or write to it, that is the slowest link in the chain. When you call PHP function file_get_contents(), the entire thread is blocked until the contents are retrieved! The server can’t do anything until your script gets the file contents. Consider what happens when multiples of simultaneous requests are issued by different users to your server? They get queued, because no thread is available to do the job since they are all blocked in I/O!

Here comes the unique selling-point of node.js. Since node.js implements async I/O in almost all its functions, the server thread in the above scenario is freed as soon as the file retrieval function (fs.readFile) is called. Then, once the I/O completes, node calls a function (passed earlier by fs.readFile) along with the data parameters. In the meantime, that valuable thread can be used for serving some other request.

So thats the theory about it anyway. But I’m not someone who just accepts any new fad in the town just because it is hype and everyone uses it. Nope, I want to get under the covers and verify it for myself. I wanted to see whether this theory holds in actual practice or not.

So I brought upon myself the job of writing two simple scripts for benchmarking this - one in PHP (hosted on apache2) and other in javascript (hosted on node.js). The test itself was very simple. The script would:

1. Accept the request.\

Generate a random string of 108 kilobytes.\
Write the string to a file on the disk.\
Read the contents back from disk.\
Return the string back on the response stream.

This is the first script, index.php:

<?php
//index.php
$s=""; //generate a random string of 108KB and a random filename
$fname = chr(rand(0,57)+65).chr(rand(0,57)+65).chr(rand(0,57)+65).chr(rand(0,57)+65).'.txt';
for($i=0;$i&lt;108000;$i++)
{
	$n=rand(0,57)+65;
	$s = $s.chr($n);
}

//write s to a file
file_put_contents($fname,$s);
$result = file_get_contents($fname);
echo $result;

And here is the second script, server.js:

//server.js
var http = require('http');    
var server = http.createServer(handler);

function handler(request, response) {
	//console.log('request received!');
	response.writeHead(200, {'Content-Type': 'text/plain'});

	s=""; //generate a random string of 108KB and a random filename
	fname = String.fromCharCode(Math.floor(65 + (Math.random()*(122-65)) )) +
		String.fromCharCode(Math.floor(65 + (Math.random()*(122-65)) )) +
		String.fromCharCode(Math.floor(65 + (Math.random()*(122-65)) )) + 
		String.fromCharCode(Math.floor(65 + (Math.random()*(122-65)) )) + ".txt";

	for(i=0;i&lt;108000;i++)
	{
		n=Math.floor(65 + (Math.random()*(122-65)) );
		s+=String.fromCharCode(n);
	}

	//write s to a file
	var fs = require('fs');
	fs.writeFile(fname, s, function(err, fd) {
			if (err) throw err;
			//console.log("The file was saved!");
			//read back from the file
			fs.readFile(fname, function (err, data) {
				if (err) throw err;
				result = data;
				response.end(result);
			});  
		}
	);
}

server.listen(8124);
console.log('Server running at http://127.0.0.1:8124/');

And then, I ran the apache benchmarking tool on both of them with 2000 requests (200 concurrent). When I saw the time stats of the result, I was astounded:

#PHP:
Concurrency Level:      200
Time taken for tests:   574.796 seconds
Complete requests:      2000

#node.js:
Concurrency Level:      200
Time taken for tests:   41.887 seconds
Complete requests:      2000

The truth is out. node.js was faster than PHP by more 14 times! These results are astonishing. It simply means that node.js IS going to be THE de-facto standard for writing performance driven apps in the upcoming future, there is no doubt about it!

Agreed that the nodejs ecosystem isn’t that widely developed yet, and most node modules for things like db connectivity, network access, utilities, etc. are actively being developed. But still, after seeing these results, its a no-brainer. Any extra effort spent in developing node.js apps is more than worth it. PHP might be still having the “king of web” status, but with node.js in the town, I don’t see that status staying for very long!

Update

After reading some comments from the below section, I felt obliged to create a C#/mono version too. This, unfortunately, has turned out to be the slowest of the bunch (~40 secs for 1 request). Either the Task library in mono is terribly implemented, or there is something terribly wrong with my code. I’ll fix it once I get some time and be back with my next post (maybe ASP.NET vs node.js vs PHP!).

Second Update

As for C#/ASP.NET, this is the most optimum version that I could manage. It still lags behind both PHP and node.js and most of the issued requests just get dropped. (And yes, I’ve tested it on both Linux/Mono and Windows-Server-2012/IIS environments). Maybe ASP.NET is inherently slower, so I’ll have to change the terms of this benchmark to take it into comparison:

public class Handler : System.Web.IHttpHandler
{
	private StringBuilder payload = null;

	private async void processAsync()
	{
		var r = new Random ();

		//generate a random string of 108kb
		payload=new StringBuilder();
		for (var i = 0; i &lt; 54000; i++)
			payload.Append( (char)(r.Next(65,90)));

		//create a unique file
		var fname = "";
		do{fname = @"c:\source\csharp\asyncdemo\" + r.Next (1, 99999999).ToString () + ".txt";
		} while(File.Exists(fname));            

		//write the string to disk in async manner
		using(FileStream fs = File.Open(fname,FileMode.CreateNew,FileAccess.ReadWrite))
		{
			var bytes=(new System.Text.ASCIIEncoding ()).GetBytes (payload.ToString());
			await fs.WriteAsync (bytes,0,bytes.Length);
			fs.Close ();
		}

		//read the string back from disk in async manner
		payload = new StringBuilder ();
		StreamReader sr = new StreamReader (fname);
		payload.Append(await sr.ReadToEndAsync ());
		sr.Close ();
		//File.Delete (fname); //remove the file
	}

	public void ProcessRequest (HttpContext context)
	{
		Task task = new Task(processAsync);
		task.Start ();
		task.Wait ();

		//write the string back on the response stream
		context.Response.ContentType = "text/plain";
		context.Response.Write (payload.ToString());
	}

	public bool IsReusable 
	{
		get {
			return false;
		}
	}
}

References:

How to create a custom email plugin for Wordpress

2014-06-07T00:00:00+00:00

Last week, I suddenly stopped receiving email notifications for my openshift hosted blog. I came to know after some reading that commonly used cloud hosts such as openshift, aws, etc. are usually blacklisted by most email servers, hence its not a good idea to use them to send mails.

In any case, why should I depend on my hosting provider for email sending. Until now, I had never bothered about how mail sending worked in wordpress as it used to work out of the box. So last week, I pulled up my socks and decided to put my php IDE and debugger to some good work.

I decided to use my sendgrid account to send mails. All that`s needed now is calling the web service with the credentials they’ve provided. But how to integrate this with my wordpress blog?

Once I located where the mail sending functionality is there in wordpress code, adding a new method was a piece of cake!! Turns out that wordpress, by default, just executes the “mail” command which is usually just a symlink on unix boxes actually pointing to /usr/bin/sendmail or something. I found it in a pluggable function wp_mail(). (see /wp-includes/pluggable.php). I also came to know from the codex that pluggable functions can be easily overridden by plugins.

Now all I had to do was write a small plugin in the /wp-content/plugins/sendgrid/ folder and override this wp_mail() function with whatever I want.

Lo and behold! I started receiving notifications for all comments and contact forms filled, by just writing this one plugin. I found the process so simple and easy to integrate with wordpress that I couldn’t help sharing with you. Here are the two php snippets that you need to place in /wp-content/plugins/your-plugin-name/ and activate it. Wordpress will do the rest!

(First one is the main plugin php file that displays the plugin in your admin menu and overrides the wp_mail function. The second php file contains the actual custom function that sends email via sendgrid.)

wp-content/plugins/sendgrid/myplugin.php:

<?php /**
 * Plugin Name: Sendgrid Plugin
 * Plugin URI:  http://prahladyeri.github.io
 * Description: Mail sending using Sendgrid Web API
 * Version:     0.1
 * Author:      Prahlad Yeri
 * Author URI:  http://prahladyeri.github.io
 * License:     MIT
 */

//namespace MailDemo;
require_once('sendgrid.php');

add_action( 'init', 'plugin_init' );

/**
 * Plugin Name: Prahlad's mail
 * Description: Alternative way to send a mail
 */
if (!function_exists('wp_mail')) 
{
	function wp_mail($to, $subject, $message, $headers = '', $attachments = array())
	{
		$sto = '';
		if (is_array($to))
		{
			$sto = implode(',',$to);
		} else {
			$sto = $to;
		}
		sendgridmail('wpadmin@mywebsite.com', $sto, $subject, $message, $headers);
	}
}

function plugin_init()
{
}

wp-content/plugins/sendgrid/sendgrid.php:

<?php //wp-content/plugins/sendgrid/sendgrid.php
function sendgridmail($from, $to, $subject, $message, $headers) {        
$url = 'https://api.sendgrid.com/';
$user='your-sendgrid-username';   
$pass='your-sendgrid-password';       
	
$params = array(       
'api_user'  ?> $user,
		'api_key'   => $pass,
		'to'        => $to,
		'subject'   => $subject,
		'html'      => '',
		'text'      => $message,
		'from'      => $from,
	  );


	$request =  $url.'api/mail.send.json';

	// Generate curl request
	$session = curl_init($request);
	// Tell curl to use HTTP POST
	curl_setopt ($session, CURLOPT_POST, true);
	// Tell curl that this is the body of the POST
	curl_setopt ($session, CURLOPT_POSTFIELDS, $params);
	// Tell curl not to return headers, but do return the response
	curl_setopt($session, CURLOPT_HEADER, false);
	curl_setopt($session, CURLOPT_RETURNTRANSFER, true);

	print_r('obtaining the response');
	// obtain response
	$response = curl_exec($session);
	print_r('closing curl session');
	curl_close($session);
	
	// print everything out
	//print_r($response);
}

//only for testing:
/*$to      = 'prahladyeri@yahoo.com';
$subject = 'Testemail';
$message = 'It works!!';
echo 'To is: ' + $to;
#wp_mail($to, $subject, $message, array() );
sendgridmail($to, $subject, $message, array());
print_r('Just sent!');*/

References:

The REAL way to make Eclipse run faster on Ubuntu

2014-06-02T00:00:00+00:00

If you are still one of those people who are frustrated with the crawling speed of eclipse IDE (especially after the recent clunky releases of Juno/Kepler), then you are in good company! Most of the advice you might have read until now for speeding up Eclipse must have involved tweaking the following parameters in eclipse.ini file:

-Xmn512m -Xms1024m -Xmx1024m -Xss2m -XX:PermSize=1024m -XX:MaxPermSize=1024m -XX:+UseParallelGC

Since Eclipse is pretty much a RAM hungry monster, feeding it with lots of RAM should surely make it run fast, right? Wrong! Until recently I had spent a lot of time tweaking those parameters, but no substantial performance could be gained (though I have 4GB of RAM with i3 which is not a bad configuration). The main issue here is that the underlying linux won’t provide the required boost to eclipse no matter whatever the parameters you provide. For instance, the system monitor shows that eclipse is only consuming 500Mb RAM, now what difference will it make if I provide 2048m to -XX:MaxPermSize?

System Monitor

My search lead me to another better approach to solving this problem. If somehow we can load JDK into a shared memory or a RAM-Disk instead of it starting from the local hard-disk, both startup time and performance could be drastically improved.

But how do we create a RAM-Disk on linux? Well, if you using ubuntu, then you are in luck! Ubuntu has a working RAM-Disk folder called /dev/shm that could be globally used by any application as a temporary storage. If you go to that folder, you can see lots of files stored by pulseaudio.

I thought why not copy the JDK folder to /dev/shm and provide that as a -vm parameter to eclipse. Lo and behold! Eclipse runs about 10 times faster on my ubuntu machine. Try it yourself and let me know (If you are having performance issues with Eclipse, that is..-).

Google appengine tip: How to clear appcfg credential cache

2014-01-19T00:00:00+00:00

Many a times, it so happens that you need to work with multiple credentials while uploading/downloading apps on Google appengine. In such a scenario, it becomes difficult to switch credentials instantly.

{.alignnone .size-full .wp-image-1382 width=”54” height=”48”}

For instance, you have just uploaded an app using appcfg.py with your google credentials and they are still stored in the cache. So when you want to upload a new app, it won’t ask you for email/password and still try to retrieve old credentials automatically ignoring any command-line parameters you have given! Thus, you keep scratching your head as to why you are getting a permission-denied error while uploading/downloading the app!

The only way out here is clearing the credential cache of appcfg. On Linux systems, these are stored in a file called:

/home/username/.appcfg_cookies

On Windows based systems, these are typically stored in:

C:\Users\username\.appcfg_cookies

Just delete this file and you are done! Next time, appcfg.py will ask you for a fresh google email and password, thus enabling you to upload/download your app.

References:

http://stackoverflow.com/q/5149914/849365

How to create a Python app in Google App Engine

2013-12-06T00:00:00+00:00

Whilst the official pythonic reference for Google app engine is the way I learnt how to build my first GAE app, I found it a bit frustrating to go through each and every link and understand large topics like caching and data stores in detail just to build a small hello world pythonic app in Google App Engine. What I wanted was a quick and dirty tutorial that let me build a small app first, and then let me improvise upon the areas that I needed to dig deeper.

{.alignnone .wp-image-301 width=”150” height=”150”}

I couldn’t find such a tutorial anywhere, so I’m writing this one.

Setup your Environment: Download and install python 2.7 for your platform, if you haven’t done so already (as of this writing, only 2.5 and 2.7 versions are supported) . Then, download and install the GAE API from here. MSI setups are available for windows platform. For linux, you can just unzip into a local folder like ‘~/programs/’. The zip file will create a subdir called ‘google_appengine’. Practically, the only two python scripts you are ever going to need to develop a GAE app are:
- dev\_appserver.py
- appcfg.py
Register your app on appspot.com: The next step is to register your subdomain on GAE by visiting https://appengine.google.com. Once you register your app there, you will get a subdomain called http://your-app-id.appspot.com. There are also options for redirecting your custom domain such as www.mydomain.com to your app subdomain.
Create your app on the local machine: This is as simple as creating a folder on your machine such as ~/source/foo in linux or C:\source\foo in windows. Then just create a text file named app.yaml with below contents inside this folder:
```
application: your-app-id
version: 1
runtime: python27
api_version: 1
threadsafe: true
	
handlers:
- url: /.*
  script: helloworld.application
```

Note that your-app-id is the name that you just registered for yourself, make sure that it is typed correctly. version parameter refers to the version of your app, while api_version is the version of GAE SDK used to run this app. The line “script: helloworld.application” indicates that this wsgi handler will be invoked for your app.

Create the wsgi handler: This is as simple as creating a python file named “helloworld.py” in the same folder as above, and add below contents to it:

import webapp2

class MainPage(webapp2.RequestHandler):

    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.write('Hello, World!')

application = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

Test your app: To test your app, open up your terminal and change directory to your GAE installation folder (alternatively, add the GAE installation folder to your PATH/$PATH environment variable to avoid doing this each time), and then type the below command:
```
python dev_appserver.py ~/source/foo
#OR on windows:
python dev_appserver.py C:\source\foo
```
Deploy your app: Want to host this app on GAE and check it out? Just fire up your terminal as described above and issue this command:
```
python appcfg.py update ~/source/foo
#OR on windows:
python appcfg.py update C:\source\foo
```
Test your app: The above command should host your app on your appspot subdomain (It will ask for your google username/password before doing so). Once the app is successfully hosted, you can check it out by visiting http://your-app-id.appspot.com.
Furthur reading: Now that you have a working app, you can actually visit the official reference to read more about:
1. webapp2: The pythonic web framework used to handle requests and generate responses.
2. Datastore: The big data storage feature that GAE provides your app to store its data.
3. Quotas and Limits: Learn about the various limits that google sets for your app to access resources (Don’t worry, they are enough to suffice a small to medium scale app).
4. App caching: Learn how to take advantage of various caching mechanisms in GAE to speed up your app.

How to create android dialogs in a reusable manner

2013-12-01T00:00:00+00:00

Creating dialogs is a very common need in your app to show a dialog box to the user in order to fetch a value, be it a mobile, desktop or even a web application. Furthermore, the values can range from anything like simple OK-Cancel dialog results to a list of “check-able” values or even a date-range.

{.alignnone .wp-image-209 width=”60” height=”70”}

I experienced the need to create an android dialog for each one of those for showing reports in a recent android app project. Whilst the java api offers maximum flexibility in creating dialog interface elements, there is no ready-made method that can be called to get, say a result for a message-dialog like this:

result = MessageBox.Show();

Other languages like C# and VB provide such methods to show modal dialog boxes that return values after waiting for a modal dialog. But unfortunately, there is no concept of “modal” in android. A thread cannot just sit idle waiting for input as the resources are too valuable for that. Instead, there is the concept of callbacks, so that instead of you waiting for the dialog to return, the method calls back a function reference you have passed it:

AlertDialog.Builder builder=new AlertDialog.Builder(context);
builder.setTitle("Milk supply tracker");
builder.setMessage(message);
builder.setPositiveButton("Yes",listener);
builder.setNegativeButton("No",listener);
builder.create().show();

The listener here is the referece to a function that will be called when the Yes or No button will be clicked. This not only complicates your code, but makes it very difficult to reuse code for handling different situations like getting a selection from a range of values or getting a date/time range. To solve this problem, I created a separate java class called Dialog and added variations of ShowDialog() methods to handle each type of dialog:

class Dialog
{
	public static void ShowMessageDialog(Context context, String message)
	{
		ShowDialog(context,message,MessageBoxType.OKOnly,new String[]{},false, null,null);
	}

	public static void ShowMessageDialog(Context context, String message, MessageBoxType type , OnClickListener listener)
	 {
		 ShowDialog(context,message,type,new String[]{},false, listener,null);
	 }

	public static void ShowListDialog(Context context, String message, String[] listItems, boolean isMultiChoice, OnClickListener listener)
	 {
	 if (isMultiChoice)
		 ShowDialog(context, message, MessageBoxType.OkCancel , listItems, isMultiChoice, listener,null);
	 else
		 ShowDialog(context, message, MessageBoxType.OKOnly , listItems, isMultiChoice, null,listener);
	 }

	public static void ShowDateDialog(Context context,String message,OnDateSetListener listener)
	{
		Calendar c=Calendar.getInstance();
		int y=c.get(Calendar.YEAR);
		int m=c.get(Calendar.MONTH);
		int d=c.get(Calendar.DAY_OF_MONTH);

		DatePickerDialog dlg=new DatePickerDialog(context, listener, y, m, d);
		dlg.setTitle(message);
		dlg.show();
	}

}

As you can see, the ShowMessageDialog() accepts different parameters depending on whether a listener is required or not. ShowListDialog(), on the other hand passes an array of strings to create a dialog displaying a list of values from which a user may select. The isMultiChoice parameter tells whether a checkbox is required or not against each value in the select list. All this is actually implemented in the ShowDialog() private method, whereas the ShowDateDialog() has its own implementation. Here is the source for ShowDialog():

private static void ShowDialog(Context context, String message, MessageBoxType type , String[] listItems, boolean isMultiChoice, OnClickListener listener,OnClickListener selectedItemListener)
{
	AlertDialog.Builder builder=new AlertDialog.Builder(context);

	if (listItems.length>0 && isMultiChoice==false)
	{
		CheckedItems=new ArrayList();//won't be used in this case.
		builder.setTitle(message);

		builder.setItems(listItems, selectedItemListener);
	}
	else if (listItems.length>0 && isMultiChoice==true)
	{
		CheckedItems=new ArrayList();
		builder.setTitle(message);

		builder.setMultiChoiceItems(listItems, null, new OnMultiChoiceClickListener() 
		{
			@Override
			public void onClick(DialogInterface dialog, int which, boolean checked) 
			{
				if (checked)
					CheckedItems.add(which);
				else
				{
					if (CheckedItems.contains(which))
						CheckedItems.remove(which);
				}
			}
		});
	}
	else
	{
		builder.setTitle("Milk supply tracker");
		builder.setMessage(message);
	}

	if (listItems.length==0 || isMultiChoice)
	{
		switch(type)
		{
		case OKOnly:
			builder.setPositiveButton("OK",listener);
			break;
		case OkCancel:
			builder.setPositiveButton("OK",listener);
			builder.setNegativeButton("Cancel",listener);
			break;
		case YesNo:
			builder.setPositiveButton("Yes",listener);
			builder.setNegativeButton("No",listener);
			break;
		}           
	}

	builder.create().show();
}

So hopefully, this class should suffice all your needs related to showing a dialog on your android app. Here is a working example of how the ShowListDialog() is actually called with checkboxes on:

selItems=new String[]{"apples","oranges","grapes"};
Device.ShowListDialog(this,"Select a fruit" ,this.selItems, true, new DialogInterface.OnClickListener() {

		@Override
		public void onClick(DialogInterface dialog, int which) 
		{
			if (which==DialogInterface.BUTTON_POSITIVE)
			{
				for(int i:Device.CheckedItems)
				Dialog.ShowMessageDialog(ReportsActivity.this, "selected:" + selItems[i]);
			}
		}
	});
}

How to uniquely identify your Android device in code

2013-11-28T00:00:00+00:00

My last android project involved tracking each device where the app is installed and storing the information to a database. It is quite a common need to uniquely identify your android device in code.

Now, had it been a PC, it would have been easy to track the MAC-address of the NIC or an HDD serial to uniquely identify and track that device. But what is the equivalent for android?

My research first led me to IMEI number (TelephonyManager.getDeviceId()). IMEI is a unique number associated with your device (pretty much like a vehicle’s chassis number) and is widely used for tracking cellphones. The android API provides this ready-made function to uniquely identify your device if you wish to go this route:

public static String getDeviceIdTm(Context context)
{
	TelephonyManager tm=(TelephonyManager)context.getSystemService(Context.TELEPHONY_SERVICE);
	return tm.getDeviceId();
}

But wait, not all devices are equipped with Telephony. What about tablets and amazon kindle devices? It so happens, that this is just one way to track your device, but it is not full-proof. It will work only for phones and for other devices this function will return null.

This led me to another way of tracking an Android device: An in-built variable that the Android system itself provides you: ANDROID_ID. In theory, this variable is all you need to know to identify your device uniquely:

public static String getDeviceIdAndroid(Context context)
{
	return Secure.getString(context.getContentResolver(),Secure.ANDROID_ID);
}

But Alas! Even this is not full-proof. It will work on most modern versions of android (HoneyComb and above). Again, due to a manufacturer bug, it will not return a unique value, but a constant value “9774d56d682e549c” on some handsets.

This led me to a third way of identifying my device which was a bit hackish. I prefer not to use this method if any of the prior two methods worked.

public static String getDeviceIdPseudo(Context context)
 {
	 String tstr="";
	 if (Build.VERSION.SDK_INT > Build.VERSION_CODES.FROYO) {
		 tstr+= Build.SERIAL;
		 tstr += "::" + (Build.PRODUCT.length() % 10) + (Build.BOARD.length() % 10) + (Build.BRAND.length() % 10) + (Build.CPU_ABI.length() % 10) + (Build.DEVICE.length() % 10) + (Build.MANUFACTURER.length() % 10) + (Build.MODEL.length() % 10);
		 return tstr;
	}
 }

This method computes a Pseudo-id for your device taking reference to some hardware values. If the previous two methods don’t work, then this is all you are left with for device identification.

I then integrated the above three methods to create a generic method called getDeviceIdUnique() that will work on all android devices - irrespective of whether its a phone/tablet or what make it is:

public static String getDeviceIdUnique(Context context)
 {
	 try {
		 String a = getDeviceIdTm(context);
		 String b = getDeviceIdAndroid(context);
		 String c = getDeviceIdPseudo(context);

		 if (a!=null && a.length()>0 && a.replace("0", "").length()>0) 
			 return a;
		 else if (b!=null && b.length()>0 && b.equals("9774d56d682e549c")==false) 
			 return b;
		 else if (c!=null && c.length()>0) 
			 return c;
		 else
			 return "";
		 }
	 catch(Exception ex)
	 {
		 return "";
	 }
 }

References:

The 7 "Bread and Butter" Plugins for your Wordpress blog

2013-11-27T00:00:00+00:00

Based on my experiments while setting up this blog, below are the 7 “bread and butter” plugins for your wordpress blog. These plugins came very handy for me and allowed me to seamlessly integrate much needed functionality in my blog without writing a single line of php code.

{.alignnone .wp-image-186 width=”77” height=”77”}

Contact Form 7 (Contact Forms): While designing this blog, a Contact page was one of my primary requirements. This plugin is popular and well maintained. Once installed, you will have a Contact tab on your dashboard that has a default contact-form with a short-code. You can copy-paste this short-code into any post or page of yours, and lo and behold! You have a contact form ready ~~such as the one on this blog~~. Of course, you can customize the default contact-form to change the fields to suit your particular needs. By default, the form filled by the user is mailed to the admin user’s email address. If you want to store the form data in the WP database in addition to that, you will have to install the Flamingo plugin along with this.
Jetpack (Multi-purpose): This is what I would term the “bread-and-butter” plugin for wordpress blog owners. Brought to you by Automattic (the company behind Wordpress), this plugin provides almost every feature a blogger can think of such as:
- Comprehensive statistics such as hits per page/post, incoming/outoing links, referrals, etc.
- Social buttons and a social-networking based comment system.
- Email subscriptions for your blog posts and comments.
- A mobile theme that automatically streamlines your site for mobile visitors.
- Allowing your readers to login to your blog using their wordpress.com account.
Akismet (Spam control): Again, this is a “bread and butter” kind of plugin. What kind of blogger, in his right mind, will trust comments on his high-traffic blog to be spam free! Akismet is a web-service that specializes in eradicating spam. Once you install this plugin, all your comments will be scanned by this web service before they make way to your dashboard.
WordPress Importer (migrating your existing wordpress blog): If you are migrating your posts from another wordpress hosted blog or a wordpress.com blog (like I did), you need this plugin to import the posts that you exported from there. In fact, wordpress automatically prompts you to install this plugin when you go to Dashboard->Tools->Import to import your posts.
XML Sitemap & Google News Feeds (Submitting sitemaps to spiders): A well written blog must also be searchable by spiders and search-engines, so that it can send readers to your blog. Now, when you submit your blog url to the search-engines like Google or Bing, it pays to have a sitemap so that the spiders know how is your blog or website structured. After installing this plugin, an automatic sitemap url is seamlessly generated for you at http://Your-WP-URL/sitemap.xml. You can then submit this sitemap url to the search engines and thus increase the visibility of your wordpress blog.
W3 Total Cache (caching and performance): Whilst I haven’t installed this plugin myself yet, it is said to improve the performance of your blog by serving static content to your users instead of running heavy-duty php scripts that increase your backend load. This plugin is also recommended by the XML Sitemap plugin.
Backup (backing up your wordpress site): You usually take an export of your posts from the Tools-Export menu, but if you want to backup your entire wordpress blog including the mysql database dump and all files used for hosting your wordpress blog, then this plugin is the way to go.

How to Generate PDFs in Python for Google App Engine

2013-11-26T00:00:00+00:00

One of my last projects based on google app engine and python involved storing form data in GAE datastore and generating PDF documents that the user can download. Whilst data storing was the easier part as google’s big data API it is pretty well documented, the trickier aspect was to convert it to PDF using python.

This was especially difficult in the face of GAE not providing an easy mechanism for disk writing that most PDF generation libraries require. To share my endeavors, I’m writing this post about how to generate pdfs in python for Google app engine.

The solution I came across was, as far as I know, the only possible way of generating PDFs in python! There are about three PDF generation utilities in python, each differing in terms of their area of usage:

Reportlab PDF library: This is the ideal library, if you want to create a pdf from scratch. It provides objects like canvas, pdfmetrics and ttfonts that help you with stuff like adding lines, shapes, images and paragraphs. This is pretty much comparable to the comprehensive iText java library or its C# port, iTextSharp. Their documentation is also good.
xhtml2pdf: If you want to simply convert an existing html document to pdf, the xhtml2pdf library comes in very handy.
pyPDF: If all you want to do is merge two PDFs page by page quickly, this library is the way to go.

I figured out after researching the above three libraries that a combination of xhtml2pdf and pyPDF is what I needed. Since I already had the html document template ready, I just put placeholders for my form data like __name__ , __occupation__, etc so that I can fill these before converting to PDF.

Now, I could fill these values from my python program, but the real challenge was storing the resulting PDF to disk, which was not allowed by google app engine! Turns out, we don’t need to actually store anything to disk. By sending the CreatePDF() output to a StringIO object, which is stored in memory instead of the filesystem, I could bypass the need to actually store anything to disk!!

f=open('template.htm','r')
sourceHtml = unicode(f.read(), errors='ignore')
f.close()
sourceHtml = template.render(tvals)
sourceHtml = sourceHtml.replace('__name__',sname)
sourceHtml = sourceHtml.replace('__address__',saddress)
sourceHtml = sourceHtml.replace('__occupation__',will.occupation)
packet = StringIO.StringIO() #write to memory
pisa.CreatePDF(sourceHtml,dest=packet)

Now, it would have been simple to just self.response.write(packet) to send this pdf download to the user, but in my case, I had to merge this generated pdf with another template-pdf which contained information like symbols, images and page-numbers that for some reason, could not be placed into the html document. So, I had to create a PdfFileReader object (coutesy of PyPDF library!), and then merge each page of my generated document with this template document. Then where do I write this merged output? Any guesses? - another StringIO object!! And then finally, write this StringIO object to self.response, so the user can download it.

packet.seek(0)
new =PdfFileReader(packet) #generated pdf
template = PdfFileReader(file("template.pdf", "rb")) #template pdf
output=PdfFileWriter() #writer for the merged pdf
for i in range(new.getNumPages()):
	page=template.getPage(i)
	page.mergePage(new.getPage(i))
	output.addPage(page)

outputStream = StringIO.StringIO()
output.write(outputStream) #write merged output to the StringIO object

self.response.headers['Content-Type'] = 'application/pdf'
fname = (will.name if mirror=='n' else will.partner)
self.response.headers['Content-Disposition'] = 'attachment; filename=' + str(fname).replace(' ','_') + '.pdf'
self.response.write(outputStream.getvalue())

Remember to add and include the below libraries before you do this:

import StringIO
import xhtml2pdf.pisa as pisa
from pyPdf import PdfFileWriter,PdfFileReader
from reportlab.pdfgen import canvas
from reportlab.lib.pagesizes import A4
from reportlab.pdfbase import pdfmetrics,ttfonts

References:

3 Steps to integrate barcode scanning in your Android app

2013-11-24T00:00:00+00:00

Whilst barcode scanning is a pretty complex and non-trivial task in itself, it could be overwhelming sometimes with android programming. Lucky for us, there exists an opensource project called ZXing (pronounced Zebra-crossing) that solves this problem for us.

The ZXing project has already done the heavy lifting by programming the core java components required to scan a 1d/2d barcode or even a PR-code in the form of a Google-play app. All you have to do is send an intent to this app and receive the scanned results that you may use in your app.

The ZXing project is Apache licensed, so it is free to use without any kind of restrictions. Follow these steps to integrate ZXing with your app:

Step 1: Download the source for IntentIntegrator.java and IntentResult.java from here and add the files to your android project sources.

Step 2: Start an intent in that part of your code where you would like to initiate the barcode scanning (such as a menu handler):

IntentIntegrator integrator = new IntentIntegrator(yourActivity);
integrator.initiateScan();

Step 3: All that remains now is to handle the result of this activity in your onActivityResult() handler. This is how I did it in my code:

if (requestCode==IntentIntegrator.REQUEST_CODE)
{
	IntentResult scanResult = IntentIntegrator.parseActivityResult(requestCode, resultCode, data);
	if (scanResult != null)
	{
		// handle scan result
		//MessageBox.run(this, "", "toString() returns: " + scanResult.toString());
	}
	else
	{
		// else continue with any other code you need in the method
		MessageBox.run(this, "", "scanResult is null.");
	}
}

That`s all! The above code not only returns the barcode scanning result to your app, but even prompts the user to install a barcode scanner app in case one isn’t there. This is the easiest and recommended way of integrating barcode scanning in your android app.

In case you want to embed the entire ZXing component in your app and don’t want to install an app separately for it, refer to the relevant links in the references section. However, this method is not recommended as your app won’t get the updates from ZXing.

References:

Some Wordpress tips and tricks

2013-11-23T00:00:00+00:00

Wordpress is a universally recognized and robust blogging platform written in the PHP language. Below are a bunch of Wordpress tricks that I’ve learned during my deployments, and I’d like to share with you.

1. Change breadcrumbs to start from “Home” instead of Site-Title: When I set up this site, the breadcrumbs in the post used to read like this:

Prahlad Yeri » wordpress » WordPress installation quick-start guide

However, I wanted it to look like this:

Home » wordpress » WordPress installation quick-start guide

In order to do that, you need to look into your theme-functions.php file and see where wordpress is echoing your breadcrumbs. In the mantra theme, I found this in the function mantra_breadcrumbs(). So accordingly, I changed get_bloginfo(‘name’) and hard-coded it to ‘Home’:

//changed by prahlad
//echo '<a href="'.get_bloginfo('url').'">'.get_bloginfo('name').' &raquo; </a>'; 
echo '<a href="'.get_bloginfo('url').'">'.'Home'.' &raquo; </a>';

You have to do this at multiple places in the mantra_breadcrumbs() (or your theme’s equivalent function) wherever get_bloginfo(‘name’) is used and replace it with ‘Home’.

2. How to crop off the big page-title that appears on top of pages: This is a nice little css trick. Just open your theme’s style.css and switch off the display for “entry-title” class elements within the “page” class elements. This will hide the page titles only on page-titles (and not posts!):

/*start: prahlad - used to hide post-titles on all pages (not posts)*/
.page .entry-title {
 display: none;
}
/*end: prahlad*/

If you notice that some whitespace is still left on top of the page, you can try some of these modifiers:

.site-header {
 padding-bottom: 0;
}

.site-content {
 margin-top: 0;
}

#content {
 margin-top: -20px;
}

3. How to add a contact-form to your post or page: In order to do that you need a Contact-Form plugin. The most popular and well-maintained plugin at that is “Contact-Forms 7”:

http://wordpress.org/plugins/contact-form-7/

This is a pretty decent form plugin that, from a single line of short-code (in your page or post), creates a basic all-purpose contact-form such as this:

Page created with Contact Forms 7

When you install this plugin, you will see an additional “Contact” tab in your dashboard. When you click that, you will see a default form with a short-code like this:

[contact-form-7 id="116" title="Contact form 1"]

You can copy-paste this short-code anywhere in the editor for your page or post. For this site’s contact page, it looked something like this:

[contact-form-7 id="116" title="Contact form 1"]

By default, the information in the text-fields are mailed to the admin user’s email-address when a user submits this form. You can change all this by editing the default form that you find in the “Contact” tab of your dashboard. You can add/remove the text-fields, add validation and much more.

Another plugin that works in combination with this is the flamingo plugin. It is useful in case you want to store the contact information in the wordpress database in addition to getting it mailed to your email address.

4: Change the footer notices (such as “Powered by Wordpress”, etc..): Just open your footer.php from editor and try to find a snippet such as this:

<footer id="colophon" role="contentinfo">
 <div class="site-info">
 <?php do_action( 'twentytwelve_credits' ); ?>

Change it accordingly to suit your needs.

References:

Wordpress installation quick-start guide

2013-11-17T00:00:00+00:00

Wordpress is a blogging platform that is very easy to use, but involves many configuration trivia which may become overwhelming, unless organized and documented somewhere. Based on my experience of setting up this website, here are the things that I had to keep in mind to get it up and running:

Wordpress file-structure: A brief understanding of the file structure helps before delving into the installation details. The folder where you upload the contents of wordpress installation files has an index.php file that triggers other configuration files from there. You have to setup the mysql database credentials in the wp-config.php file before starting the installation. Inside the root folder, there are two sub-folders, wp-admin and wp-content. The wp-admin folder stores the necessary files to bring up the admin interface when you type http://your-wp-url/wp-admin to administer your site for adding posts, moderating comments, etc. The wp-content folder on the other hand, stores contains files and folders to create the structure, functionality and look and feel of your blog. The wp-content folder further has wp-themes and wp-plugins folders. The former controls the structure and look and feel (themes), whereas the latter is for adding extra bits of extra functionality (plugins).
- root-folder » index.php, wp-config.php, wp-settings.php
  - wp-admin - Files & folders to control admin interface.
  - wp-content
    - wp-themes - Files and folders to control look and feel.
    - wp-plugins - Files and folders to add bits of extra functionality.
  - wp-includes - Contains header files for other php scripts.

Famous File-minute installation:Once you have the basic understanding, you may proceed with the much advertisedfamous five minute installation of your wordpress site. Basically, once you have set the mysql credentials in wp-config.php, wordpress does the rest by creating the required tables for your blog-posts, comments and other elements. But make sure, the wp-user you configure has suffient rights to create/alter/query/etc. on the wordpress database you have configured.
Configuration: These settings in wp-config.php come quite handy during deployment:
- DB_NAME - mysql database name reserved for the wordpress blog.
- DB_USER - mysql user-id.
- DB_PASSWORD - mysql password.
- DB_HOST - hostname of the machine where mysql is running.

The above settings are almost always different in your local environment from that of your web-hosting machine. Hence, it is advisable that you keep the local version of wp-config.php separate by adding an exception to version control systems like git, svn, etc. if you are using any.

Site name settings: Two php global variables called WP_HOME and WP_SITEURL are almost always a source of trouble during migration of wordpress sites. These variables tell wordpress the url of your currently hosted wordpress site (such as http://localhost/wp or http://www.mysite.com/blog). When you migrate your database settings to your web host, wordpress obviously can’t find the localhost there. To temporarily solve this issue, you can hard-code these values at the start of wp-config.php like this:
- define(‘WP_HOME’,’http://workstation2/rhc/prahladyeri/php’);
- define(‘WP_SITEURL’,’http://workstation2/rhc/prahladyeri/php’);

Once you have access to wp-admin, you should remove them from wp-config.php and go to Settings->General to set these variables in a proper manner under the fields, “Wordpress Address (URL)” and “Site Address (URL)”.

Prahlad Yeri

Readymade template engine using the function constructor trick in JavaScript

Comparison Table

How AI can turn us into a society of p-zombies

I am a programmer, not a rubber-stamp that approves Copilot generated code

Unpopular Opinion: Implicit Usings are an Anti-Pattern

1. Readability suffers

2. Potential for ambiguity

3. Convenience should not trump maintainability

My preferred approach

Create a C# Windows Desktop App in 9 Lines — No Visual Studio Needed

Step 1: Create the source file

Step 2: Compile and run

What’s next?

The art and science of effectively maintaining changelogs

Stoicism vs Nihilism: The Power of Choice

When Gandhi met Satoshi

Gandhi (smiling gently):

Satoshi (nodding):

Gandhi:

Satoshi:

Gandhi:

Satoshi:

Gandhi (thoughtfully):

Satoshi (softly):

Gandhi (smiling):

How to revive a Micromax Canvas Lapbook L1161 in 2025

A Note on Linux Compatibility

Final Thoughts

The ultimate guide to boosting productivity with the Pomodoro Technique

What is the Pomodoro Technique?

Why does the Pomodoro Technique work?

Setting up for success

1. Choose the right tools

2. Plan your tasks

3. Eliminate distractions

Advanced tips for seasoned users

1. Customize Pomodoro lengths

2. Batch similar tasks

3. Analyze your productivity data

Potential challenges and solutions

Real-world success stories

Beyond productivity: Unexpected benefits

Final thoughts

Master PDF digital signing with eMudhra and Proxkey in .NET: step-by-step guide

Preparing the environment.

Setting up the Project

Signing the PDF with Proxkey

Getting the thumb print

Self-signed certificate

Notes on legal and regional compliance

Building a secure and efficient MQTT-HTTP gateway with Node.js

Preparing the Node.js environment.

Optional: CORS package

Setting up your MQTT brokers

Initializing middleware for HTTP requests

Establishing MQTT connections

Bridging MQTT with Express API routes

Starting the Express server

Building a simple customer management system in PHP with MySQL

What is a customer management system?

Prerequisites

Step 1: Setting up the project

1.1 Install and configure your server

1.2 Create the project folder

Step 2: Designing the database

2.1 Create the database

2.2 Create the customers table

Step 3: Building the front-end

3.1 Creating the HTML layout

Step 4: Writing the back-end logic

4.1 Database connection

4.2 Fetch and display customers

4.3 Adding a customer

Step 5: Styling the app

Step 6: Adding update and delete functionality

Update

Delete

Conclusion

References

2.2 Create the `customers` table