Abhishek Tiwari

Securing FastAPI Applications with JWT Tokens and OAuth2 using axioms-fastapi

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 30 Nov 2025 21:41:00 +0000

Securing FastAPI applications using OAuth2 doesn’t have to be complex. No matter which OAuth2 authorization server you use - Auth0, AWS Cognito, Okta, Microsoft Entra, or Keycloak - axioms-fastapi provides production-ready security for your FastAPI application. This tutorial covers how to implement authorization in your FastAPI applications using claim-based fine-grained authorization through scopes, roles, and permissions including role-based access control (RBAC), and row-level security (object-level permissions) using JWT Tokens issued by a trusted OAuth2 compliant authorization service.

JWTForge: A JWT Vending Service for Testing, Fuzzing, and Security Research of OAuth2/OIDC Implementations

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 15 Nov 2025 23:46:00 +0000

Summary

JWTForge is a lightweight, open-source JSON Web Token (JWT) (RFC 7519) vending service designed specifically for developers and security researchers who need to test or surface vulnerabilities in the OAuth2 (RFC 6749) and OpenID Connect (OIDC) (see here) implementations. With one-click, JWTForge can be deployed on Cloudflare Workers (see here), and it generates realistic, customizable JWT tokens on-demand, making it an essential tool for end-to-end testing, security auditing, and fuzzing authentication systems. Whether you’re building a new API, conducting penetration tests, or debugging authentication flows, or researching new attack modes, JWTForge provides the flexibility to generate any token configuration you need - from standard OAuth2/OIDC claims to malformed tokens for edge-case testing.

Giving Second Life to a Scientific Software: Introducing Hydrogen Bond Analysis Tool (HBAT) v2

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 30 Jun 2025 17:57:00 +0000

Every so often, we get a chance to look back at our past work. Sometimes it’s with a cringe, sometimes with nostalgia, and occasionally, with an opportunity to rebuild. During a recent and much-needed time off, I found myself embarking on a project that was a blend of all three: a complete, ground-up rewrite of the Hydrogen Bond Analysis Tool (HBAT), a piece of software I first wrote nearly two decades ago.

FREE FAANG Mock Interview

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 10 Mar 2025 05:16:48 +0000

FREE FAANG Mock Interview

1-hour mock interview with detailed analysis and feedback. We can cover System design, TMP Core, Product Management Core, Tech Depth, Operational Excellence, Leadership Principles, Behavioural, Tech Leadership, People Leadership from a hiring managers prospective. Please advise during booking what you want to focus during the mock interview. Share your CV or LinkedIn profile in booking request.

Mentoring

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 10 Mar 2025 05:16:48 +0000

Mentoring

I am mentoring at ADPList. Please book one of the available sessions for free. As token of appreciation please feel free to write a feedback at ADPList and LinkedIn.

The Hidden Cost of Success: Understanding Non-Fatal Errors in Microservices

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 28 Dec 2024 17:07:00 +0000

A recent study published by researchers from Washington University in St. Louis and Uber Technologies reveals a critical but often overlooked pattern in microservices architecture: non-fatal errors. While these errors do not cause system failures, they introduce noticeable performance overhead that can impact user experience and operational costs. The study by Lee et al. explores the prevalence of non-fatal errors and their impact on the observed latency of top-level requests.

Pitch your computer science research

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 28 Dec 2024 05:16:48 +0000

The Art of Standing Out

Are you a computer science researcher? Your research work deserves more than just publication – it needs to reach to the right audience. If you have recently published an interesting computer science research paper, share with us ➤ here. We will publish a review here on this website and potentially invite you to our upcoming Infinite Monkey podcast. More importantly, there is no charge for review published on this website or appearance on our podcast - its completely free. We will be more than happy to provide our perspective on your research findings to media outlet through quotes.

Distributed Traces of Microservices Architecture: Publicly Available Datasets

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 27 Dec 2024 10:48:00 +0000

Distributed traces are essential for understanding the behaviour, performance, and reliability of microservices architecture. They can be used to surface meaningful observations about service dependencies, call graphs, and runtime dynamics, enabling software engineers and scientists to develop new tools for optimisation and fault diagnosis. However, there is a clear gap in terms of publicly available real-world distributed tracing datasets of microservices architecture to validate theoretical research with actual observations.

Power Laws and Heavy-Tail Distributions in Hyperscale Microservices Architectures

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 22 Dec 2024 13:04:00 +0000

Recent analyses of Meta and Alibaba’s production microservices architectures identified patterns of heavy-tail and power law distributions. These patterns manifest in service scale, and request patterns, providing a glimpse into the inherent characteristics of large-scale distributed systems.

Microservice Architecture of Alibaba

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 22 Dec 2024 11:43:00 +0000

Alibaba has built a complex system of microservices to support its large user base and manage its diverse business operations. This article explores key learning from Alibaba’s microservices architecture, presenting critical observations from its design, scalability, performance optimisation, and resource allocation model. The article draws from research published by Luo et al., which characterises the structural properties of call graphs within Alibaba’s large-scale microservices ecosystem, uncovering several graph-based properties and insights (see here and here).

Unveiling Graph Structures in Microservices: Service Dependency Graph, Call Graph, and Causal Graph

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 21 Dec 2024 12:14:00 +0000

The rise of service-oriented architecture (SOA) and microservices architecture has led to a major shift in software development, enabling the creation of complex, distributed systems composed of independent, loosely coupled services. These architectures offer numerous benefits, including scalability, flexibility, and resilience. However, the distributed nature of these systems introduces new challenges related to understanding, managing, and analysing their behaviour. Graph-based modelling has emerged as a powerful technique for addressing these challenges. Graphs provide a natural and intuitive way to represent the relationships between different components of an SOA or microservices system, allowing developers and operators to gain insights into the system’s structure, dependencies, behaviour, and performance.

Cache Me If You Can: Taming the Caching Complexity of Microservice Call Graphs

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 17 Dec 2024 00:57:00 +0000

As microservices architectures have become increasingly common in modern software systems, they have brought benefits and challenges. One of the most pressing challenges has been maintaining performance at scale while dealing with complex service dependencies and network communication overhead. Today, we want to explore MuCache, a caching framework recently presented by Zhang et al. at USENIX NSDI 2024 that tackles this challenge head-on by providing automatic and coherent caching for microservices call graphs (see here).

Privacy Engineering: Why Standards Are Still Out of Reach?

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 12 Dec 2024 00:20:00 +0000

In contemporary technology environments, organisations are increasingly challenged with the complexities of privacy engineering. The evolving data governance and regulatory ecosystems demand not only technical ingenuity but also a deep understanding of legal frameworks and organizational dynamics. The role of the privacy engineer, though still emerging, sits at intersection, translating abstract privacy principles and compliance mandates into tangible technical controls and meaningful organisational practices. The article covers research findings reported by Zachary Kilhoffer and Devyn Wilder from University of Illinois at Urbana–Champaign in their recent presentation in 2024 USENIX Conference on Privacy Engineering Practice and Respect(see here). Their research findings derived from in-depth interviews with experienced privacy engineers, providing insight into their responsibilities, the ways in which they engage with standards and frameworks, and the challenges that shape this emerging profession.

Policy Zones: Purpose Limitation at Scale using Information Flow Control

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 11 Dec 2024 22:35:00 +0000

At the heart of privacy lies the principle of purpose limitation, dictating that data should only be processed for explicitly stated purposes. This principle presents a considerable challenge, especially for organisations operating at the scale of Meta, which handles vast amounts of data from billions of users. This article examines Policy Zones, a core component of Meta’s Privacy Aware Infrastructure (PAI), designed to address the complexities of purpose limitation at scale. This review is based on notes from a recent 2024 USENIX Conference on Privacy Engineering Practice and Respect(PEPR'24) presentation by Rituraj Kirti and Diana Marsala from Meta (see here).

Deploying Hugo Sites on Cloudflare Pages with Decap CMS and GitHub Backend

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 03 Dec 2024 22:26:00 +0000

Recently I migrated this website from Ghost to Hugo. This site is now generated by Hugo, stored by Github, deployed on Cloudflare Pages, and content managed via Decap CMS. Hugo, Decap CMS, Cloudflare Pages, and GitHub together create a powerful and efficient stack for building, managing, and deploying static websites. This combination is perfect for anyone seeking to develop a scalable and flexible publishing workflow, as well as content creators looking for an easy-to-use content management interface.

Differential Privacy: 6 Key Equations Explained

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 01 Dec 2024 03:16:42 +0000

Differential Privacy is a powerful framework for ensuring privacy in data analysis by adding controlled noise to computations. Its mathematical foundation guarantees that the presence or absence of any individual’s data in a dataset does not significantly affect the outcome of an analysis. Here are six key equations that capture the essence of differential privacy and its mechanisms, along with references to their origins and explanations.

Archiving with The Rogue Scholar

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 01 Dec 2024 00:07:30 +0000

Since 2009, I have published over 125 articles on this blog, creating a space where my work intersects with the rigor of academic research and the applied challenges of industry. This platform has grown beyond a personal archive. It has become a resource for researchers, industry practitioners, and students alike. It currently attracts more than 18k monthly visitor from across the globe. My articles have been frequently cited in peer-reviewed journals, conference proceedings, technical standards, patents, and academic theses.

Input vs Output Privacy

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 03 Nov 2024 02:57:20 +0000

Privacy in data systems has traditionally focused on protecting sensitive information as it enters a system - what we call input privacy. However, as systems become more complex and capable of inferring sensitive information from seemingly harmless data, the importance of output privacy has gained significant attention. Let’s explore these two crucial aspects of privacy protection and understand how different techniques address them.

Birds of a Feather Leak Together: The Set Bias Privacy Problem

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 24 Oct 2024 02:13:37 +0000

Secure multi-party computation (SMPC) enables organisations to collaborate on sensitive data analysis without directly sharing raw information. However, seemingly harmless aggregate outputs, particularly private set intersection (PSI), can leak individual-level information when analysed strategically over time. This post is based on research presented by Guo et. al at the 31st USENIX Security Symposium and examines how privacy systems can be vulnerable to sophisticated inference attacks that exploit natural dataset biases. This post is also a nice continuation from previous post where we covered privacy-preserving multi-touch attribution case study from TikTok(see here) and briefly discussed “differential attacks” exploiting gaps in PSI protocols.

Privacy-Preserving Multi-Touch Attribution at TikTok

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 22 Oct 2024 02:49:36 +0000

Multi-touch attribution is considered as holy grail in advertising industry. As advertisers are targeting users with multiple advertisements across different platforms and publishers, understanding how each of these touch points contributes to conversion is crucial—but this understanding has traditionally come at the cost of user privacy. In a previous blog post(see here) we briefly covered Mozilla’s Private Attribution API(see here) that aims to provide advertisers with conversion data without compromising user privacy. In recent PEPR ‘24 Conference, TikTok team talked about how they are integrating Differential Privacy (DP) into their ad measurement systems (see here). This blog post is notes from the talk.

The Where's Waldo Effect in Privacy

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 20 Oct 2024 03:58:46 +0000

Safeguarding individual privacy inherently means data minimisation i.e. limiting the collection and disposal of data. This principle has been a cornerstone of privacy advocacy and is even enshrined in regulations like the EU’s General Data Protection Regulation (GDPR). However, a research published by Ponte et. al (see here) is challenging this fundamental assumption, introducing what they call the “Where’s Waldo effect. They demonstrate a counterintuitive relationship between sample size and customers’ privacy risk (at least in certain scenarios).

Exploring Homomorphic Encryption with Python

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 19 Oct 2024 22:29:08 +0000

Homomorphic encryption is a powerful cryptographic technique that allows computations to be performed on encrypted data without decrypting it first. This blog post will introduce the concept of homomorphic encryption and demonstrate implementations using Python.

Top-down vs. Bottom up Privacy

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 17 Oct 2024 00:05:37 +0000

Tech companies and large consumer businesses are grappling with how best to protect end-user data while maintaining pace of innovation and competitive edge. Two distinct approaches have emerged: top-down and bottom-up privacy. Understanding these approaches is essential for anyone involved in privacy engineering, product development, or driving tech policy decisions. In this blog post, we will deep dive into the nuances of each approach, examine their strengths and weaknesses, contrast aspects of privacy such proving compliance, auditing, and monitoring.

Distributed Aggregation Protocol (DAP) Primer

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 02 Oct 2024 01:20:03 +0000

In last post we covered, Privacy Preserving Measurement (PPM) and discussed how Distributed Aggregation Protocol (DAP) works (see here). Today, we’ll explore how to implement a simplified version of the DAP using Python with Prio3 as our Verifiable Distributed Aggregation Function (VDAF). This implementation will support multiple clients, demonstrating how DAP can aggregate data from multiple sources while maintaining privacy. Let’s dive in!

Privacy Preserving Measurement

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 01 Oct 2024 03:55:06 +0000

In 1982, Andrew Yao proposed the Millionaire Problem which discusses how two millionaires can learn who is richest one without disclosing their actual wealth. They solve this problem by comparing their wealth using secure two party computation to ensure that they learn only the richest one and nothing else is revealed. The problem was later generalised for secure multiparty computation by Goldreich et al in 1987.

Managing Differential Privacy in Large Scale Systems

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 22 Sep 2024 03:24:05 +0000

The promise of differential privacy is compelling. It offers a rigorous, provable guarantee of individual privacy, even in the face of arbitrary background knowledge. Rather than relying on anonymization techniques that can often be defeated, differential privacy works by injecting carefully calibrated noise into computations. This allows aggregate statistics and insights to be extracted from data while masking the contributions of any single individual. Major players like Apple, Google, and the US Census Bureau have already adopted differential privacy for various applications. As privacy regulations like GDPR and CCPA raise the stakes for data protection, we’re likely to see differential privacy become increasingly mainstream.

Differential Privacy: A Primer

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 22 Sep 2024 01:32:31 +0000

Differential Privacy (DP) is a mathematical framework that protects individual privacy in data analysis while allowing useful insights to be extracted. It works by adding carefully calibrated noise to data or query results, ensuring that including or excluding any single individual’s data doesn’t significantly change the analysis outcomes. This approach makes it extremely difficult to infer information about specific individuals, providing a formal guarantee of privacy (see here). Differential privacy is widely used by companies, researchers, and government agencies to analyze sensitive information such as census data, medical records, and large-scale user behavior while preserving privacy.

Mathematical Guarantee

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 19 Sep 2024 02:20:30 +0000

A mathematical guarantee is a formal, provable assurance about the behavior, performance, or properties of a system, algorithm, or process, derived from rigorous mathematical analysis or proof. The guarantee is based on mathematical logic and can be demonstrated through mathematical reasoning. It provides a definitive statement about what can be expected under specified conditions.

Engineering excellence is how NASA flawlessly landed Curiosity on Mars

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 15 Sep 2024 19:31:27 +0000

We’ve all experienced the frustration of our smartphones crashing or freezing at the worst possible moment. So how is it that NASA can land a rover on Mars, millions of miles away, with software that works flawlessly? The answer lies in a combination of factors that set space-grade software apart from your average app. Let’s dive deep into the world of aerospace software engineering and uncover the secrets behind NASA’s incredible reliability.

Engineering excellence: the art of doing small things really well

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 15 Sep 2024 18:48:23 +0000

Recently I have been reading Art of Leadership, The: Small Things, Done Well by Michael Lopp. This book is an excellent read and it covers small things that you can do to build trust and to become an authentic and true leader to your team at different stages of your leadership journey. Michael use stories from his tenure as manager at Netscape , as director at Apple , and as executive at Slack to capture and distill a list of wisdom to help anyone be effective at management and leadership. Book advocates that by focusing on the small things and executing them well, leaders can create a lasting impact on their teams and organisations. It argues that great leaders understand that success is built on a foundation of countless small, well-executed actions.

The funnel of relevance

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 15 Sep 2024 17:37:13 +0000

Anyone who has worked with advanced search systems knows the intricacy of boolean operators. AND, OR, NOT - these seemingly simple words become powerful tools in the hands of search specialists. Add in proximity operators like WITH and NEAR, throw in some parentheses for nesting, and you’ve got yourself a query that can pinpoint exactly what you’re looking for… or so we thought.

What is worth of your data to LLM?

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 08 Sep 2024 13:32:05 +0000

When a machine learning model is trained on a dataset, not all data points contribute equally to the model’s performance. Some are more valuable and influential than others. Unfortunately value of data for training purposes is often nebulous and difficult to quantify. Applying data valuation to large language models (LLMs) like GPT-3, Claude 3, Llama 3.1 and their vast training datasets has faced significant scalability challenges to date. We need principled and scalable ways to answer data valuation related questions.

Microservice Architecture of Meta

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 08 Sep 2024 03:36:16 +0000

Microservices have become the dominant architectural paradigm for building large-scale distributed systems, but until now, their inner workings at major tech companies have remained shrouded in mystery. A recent new paper by Huye et. al - researchers at Tufts University and Meta - provides an unprecedented look under the hood at Meta’s massive microservices architecture (see here).

Modelling data flows as graphs to apply user privacy constraint

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 08 Sep 2024 01:41:43 +0000

Personal data processing forms the backbone of many big tech service providers. Tech giants like Netflix, Meta, and Amazon employ intricate networks of microservices to automatically process user data, creating complex data flows that span multiple layers of computation. Privacy concerns and regulatory frameworks like the General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), Brazilian Lei Geral de Protec ̧a ̃o de Dados (LGPD), Digital Markets Act (DMA) mandate that users should have control over usage and processing of their data. Users can opt out of certain types of data processing. These user-imposed constraints can significantly impact the service provider’s utility.

Kernel of a good strategy - Why, How and What

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 01 Oct 2023 16:25:45 +0000

Having a well-defined strategy is crucial for success of a modern business. A good strategy should address the core problems or challenges faced by a business, provide guiding principles on how to tackle them, and be supported by the coherent actions and right resources. In this article, we will explore the core elements of a good strategy. We will also cover a specific example of a Software-as-a-Service (SaaS) Media Intelligence (MI) business that faced significant challenges and had to reinvent its product offerings to meet customer demands.

Pre-Deployment Policy Compliance

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 23 Jul 2023 01:34:47 +0000

The ability to deploy applications quickly and efficiently is crucial for organizations. Agile methodologies and continuous integration/continuous deployment (CI/CD) pipelines have become the norm, enabling rapid release cycles and frequent updates. However, amidst the drive for speed, ensuring policy compliance is often overlooked, leading to potential security vulnerabilities and compliance risks. Pre-deployment policy compliance, supported by policy as code frameworks such as Sentinel, Open Policy Agent (OPA), Conftest, etc. is an emerging approach that enables organizations to ensure secure deployments while adhering to policies and regulatory requirements. In this article, we will explore into the significance of pre-deployment policy compliance, explore the role of policy as code frameworks, and showcase real-life examples of successful implementations.

Sufficiently Advanced Monitoring is Indistinguishable from Testing

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 08 Jul 2023 14:40:05 +0000

In digital landscape user experience is paramount and organisations must ensure their applications and systems are functioning optimally at all times. Traditional testing and monitoring practices have long been relied upon to achieve this goal. However, the line between testing and monitoring is becoming increasingly blurred with the emergence of synthetic monitoring.

Data Pipelines: The Hammer for Every Nail

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 08 Jul 2023 03:21:09 +0000

For big data and complex data processing, data pipelines have emerged as a popular solution for managing and manipulating data. They provide a systematic approach to extract, transform, and load (ETL) data from various sources, enabling organizations to derive valuable insights. However, as with any technology trend, data pipelines have not been immune to misuse and overuse. In this article, we explore the idea that data pipelines have become the new anti-patterns, as they are often misapplied and misunderstood. I have written my thoughts about this topic a few years back but not a lot changed since then (see here).

Using Inversion as a Mental Model for Product Management

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 07 Jul 2023 00:23:25 +0000

Ability to think critically and anticipate potential problems is essential for success for product management. One mental model that can greatly aid product managers in this regard is inversion. Inversion, a concept popularized by the German mathematician Carl Jacobi, involves thinking about a problem in reverse order or considering the opposite of what is desired. Inversion is a powerful technique that involves flipping a problem on its head and considering the opposite perspective. By envisioning the negative outcomes or obstacles that may arise, product managers can proactively address them and create more robust and successful products. Here’s how you can use inversion as a mental model for product management.

Roles vs Jobs

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 27 Apr 2021 12:05:17 +0000

It seriously turns me off when organisations are unable to differentiate between a role and a job. Talking about this confusion, Scrum Product Owner probably one of the most misunderstood roles these days which unfortunately many organisations advertise as a job.

Continuous Delivery of Games

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 08 Jan 2021 03:30:09 +0000

What can we learn from Cyberpunk 2077’s disastrous launch? Cyberpunk 2077 was extensively advertised yet highly buggy release. Sony has pulled Cyberpunk 2077 from its PlayStation store following widespread complaints about glitches. The game lost over 75% of players on Steam. The stock value of CD Projekt Red (CDPR) - Developer of Cyberpunk 2077 - is down by 50%. CDPR board and management are now defensive as they now face hostile staff (i.e. overworked software engineers) after the failed launch.One key learning for me is the lack of continuous delivery and modern software engineering practices in the gaming industry.

Organizational debt

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 08 Jan 2021 02:46:50 +0000

Organizational debt is a sibling of technical debt. While technical debt is unhealthy and generally a well-understood problem, organizational debt can be killer and often treated as an afterthought. The term “organizational debt” was coined by Steve Blank, defined as the accumulation of changes that leaders should have made but didn’t. Things like biased and subjective hiring process, poorly defined roles and job descriptions, gender compensation gaps, lack of diversity and inclusion, delay in addressing performance issues, tolerating brilliant jerks far too long, outdated or badly designed policies, using external recruiters are just a few examples of organizational debt.

Introducing Axioms

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 27 Feb 2020 03:18:39 +0000

I am thrilled to announce the early availability of Axioms - a financial-grade identity and trust management platform. Axioms enables organizations to protect their apps, devices, and APIs against unauthorized and untrusted access using our fully-managed SaaS offering. We believe that identity and trust are two major blind spots in cybersecurity and we are on the mission to fix it using our standard-based platform assisted with the powerful built-in automation and machine learning smarts.

Your technology architecture and engineering organization should coevolve as your startup grows

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 27 Feb 2020 02:35:01 +0000

The evolution of your technology architecture should depend on the size, culture, and skill set of your engineering organization. In fact, they should coevolve. There are no hard-and-fast rules to figure out interdependency between technology architecture and engineering organization but below is what I think can really work well for a typical product startup.

Partnership over Stakeholdership

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 22 Jul 2019 10:53:39 +0000

When hiring for senior roles, companies put too much stress on stakeholder management skills i.e. ability to navigate through complex stakeholder landscape and manage expectations. I find the ability to form meaningful and impactful partnership a lot more important skill than stakeholdership. So what defines partnership at the workplace and how you use it to hire the people who can create great partnerships and have a long-lasting impact on the course of your company?

Organise your engineering teams around the work by reteaming

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 20 Jul 2019 11:06:48 +0000

When it comes to organising engineering teams, a popular view has been to organise your teams based on either Spotify’s agile model (i.e. squads, chapters, tribes, and guilds) or simply follow Amazon’s two-pizza team model. On a positive note, both organisational models focus on having small yet productive independent cross-functional teams and there is nothing wrong to draw inspiration from either of these models when building your engineering organisation. Both models work on the scale, particularly when the company’s core values, priorities, and culture all three are well aligned. Now, this is where most companies fail, either by putting too much focus on core values and culture but forgetting about business priorities or just focusing on business priorities but not enough stress on core values and culture.

Agile: Optimise for velocity over predictability

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 09 Jan 2019 12:18:16 +0000

When it comes to agile estimation, quite often agile teams spend a big chunk of their time in heavyweight processes like detailed story point estimation to improve the predictability or accuracy of their estimates. In other words, agile teams invest more time in estimation to reduce that variance (i.e. time estimated vs. time taken) which otherwise can be used to create features and add business value. In a nutshell, they are slowing down to be more predictable. Unfortunately, it’s not good enough to be predictably slow.

Content Management Systems of the Future: Headless, JAMstack, ADN and Functions at the Edge

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 03 Nov 2018 10:37:34 +0000

Recently I was asked about content management systems (CMS) of the future - more specifically how they are evolving in the era of microservices, APIs, and serverless computing. For enterprise customers either undergoing or planning the digital transformation, this is an important question to ask. Enterprise customers spend a large chunk of their digital and marketing budget on CMS and associated modules such as digital asset management (DAM). Most of the CMS vendors dodge questions of evolution by talking about incremental innovation primarily focused on customer experience (CX) such as analytics and personalisation. Unfortunately, due to the lack of a good developer experience (DX) total cost of ownership of a traditional CMS and implementation failure rate remains all-time high. Strictly speaking, it cost companies more to implement and maintain a CMS than just CMS license fee. To sum it up, if anything CMS landscape now requires a big shift. Well, you don’t have to wait, CMS of the future is already here - it is headless, static, distributed, cost-effective with equal focus on CX and DX.

Friends don't let friends build data pipelines

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 12 Jul 2018 10:38:05 +0000

Building data pipelines can offer strategic advantages to the business. It can be used to power new analytics, insight, and product features. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines.

Why engaged teams are not always the best performers?

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 08 Jul 2018 13:41:26 +0000

The assumption that engagement is somehow strongly correlated with performance is misplaced. Traditional engagement metrics such as satisfaction, happiness, well-being let alone cannot explain behaviours, actions and motivation of a high-performance team. Moreover, there is hardly any strong scientific evidence suggesting that there is a correlation between high-engagement and high-performance. Given the similar type of environments, two teams with the same level of engagement may have different levels of performance - one can be high-performing another average performing. A high-performing team can be highly engaged but vice verse is not always true. In the context of engineering teams, high-performing teams usually show a bit of discontent and frustration which the traditional model of engagement (e.g. a happy employee is a productive employee) will be unable to capture or explain.

ETL Workflow Modeling

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 14 Apr 2018 23:30:58 +0000

Developing Extract–transform–load (ETL) workflow is a time-consuming activity yet a very important component of data warehousing process. The process to develop ETL workflow is often ad-hoc, complex, trial and error based. It has been suggested that formal modeling of ETL process can alleviate most of these pain points. Generally speaking, formal modeling can reduce implementation time and save money by adopting structural patterns and best-practices when implementing ETL workflows.

10 open-source Kubernetes tools for highly effective SRE and Ops Teams

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 06 Jan 2018 13:00:00 +0000

If you are running workloads in Kubernetes, your site reliability engineering (SRE) and operations (Ops) teams need right kind of tooling to ensure the high-reliability of the Kubernetes cluster and workloads running in it. Here we present a list of 10 open-source Kubernetes tools to make your SRE and Ops teams more effective to achieve their service level objectives.

5 data integration trends that will define the future of ETL in 2018

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 28 Dec 2017 01:24:00 +0000

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in last decade. With the arrival of new cloud-native tools and platform, ETL is becoming obsolete. There are several emerging data trends that will define the future of ETL in 2018. A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures.

Kubernetes for Big Data Workloads

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 27 Dec 2017 11:23:36 +0000

Kubernetes has emerged as go to container orchestration platform for data engineering teams. Kubernetes has a massive community support and momentum behind it. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads¹² and data workloads are up next. Kubernetes has been particularly successful with microservices and contemporary applications. In fact, if we go by current trends containerised microservices running on Kubernetes are the future.

Getting started with Conduit - lightweight service mesh for Kubernetes

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 25 Dec 2017 12:31:15 +0000

On this blog from very early on, we have advocated the concept of service mesh. In fact, our post a sidecar for your service mesh is one of the most viewed posts this year. When Buoyant announced the Conduit - their next-generation lightweight service mesh for Kubernetes - we were really thrilled. Buoyant is also the creator of Linkerd which is one of the most widely used service mesh currently available to the microservices community. Linkerd can already run on Kubernetes, Mesos, cluster of hosts.

How machine learning is accelerating data integration?

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 24 Dec 2017 01:12:35 +0000

Data integration generally requires in-depth domain knowledge, a strong understanding of data schemas and underlying relationships. This can be time-consuming and bit challenging if you are dealing with hundreds of data sources and thousands of event types (see my recent article on ELT architecture). Various data integration solution providers are trying to capitalize on this gap by offering various machine learning based features to overcome these challenges.

A case for ELT

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 22 Dec 2017 22:32:50 +0000

A case for ELT (i.e. extract, load, and transform) and difference between ETL and ELT

Upcoming of the learned data structures

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 14 Dec 2017 04:07:57 +0000

Can machine learning-based data structures i.e. learned data structures replace traditional data structures? This is a question recently asked and explored by a team of Google researchers led by Jeff Dean with a major focus on database indexes. Jeff is a Google Senior Fellow in the Google Brain team and widely known as a pioneer in artificial intelligence (AI) and deep learning community. The initial results show that learned data structures can outperform existing data structures. Obviously, Jeff and team don’t claim to completely replace traditional data structures with learned data structures, but their seminal work opens up an entirely new direction for the research in this space.

Rise of TrueTime: Rationale behind Amazon Time Sync Service

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 01 Dec 2017 11:13:36 +0000

At re:Invent 2017, Amazon Web Services (AWS) announced Amazon Time Sync Service which provides a highly accurate and reliable global time reference. This service is immediately available in all AWS regions.

What I've learned so far about React

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 25 Sep 2017 15:04:13 +0000

Over the weekend, I took a shot to build something new in React. React is a JavaScript library to build user interfaces. React was open sourced by Facebook and since then it has gained popularity over other frontend frameworks such as Angular, Vue, etc. To start, I have a good experience in JavaScript and I have built several applications using Node, Express, and Angular. Unfortunately, React landscape is way too complex. It is quite a different ecosystem and a difficult one for beginners.

Local Development Environment for Kubernetes using Minikube

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 10 Sep 2017 22:23:00 +0000

Kubernetes can be an ultimate local development environment particularly if you are wrangling with a large number of microservices. In this post, we will cover how you can create a local development workflow using Minikube and tools such as Make to iterate fast without the wait imposed by your continuous integration pipeline. With this workflow, you can code and test changes immediately.

A playbook for leaders in engineering

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 30 Aug 2017 09:17:08 +0000

A playbook for software engineering leaders on how to build, grow and run high-performing engineering teams.

Presentations

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 29 Aug 2017 11:00:15 +0000

For a complete list of presentations see my Speaker Deck and Slideshare profiles.

Leadership as a service

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 29 Aug 2017 07:16:10 +0000

Leadership as a service is not about extracting work or anything from teams but it’s about serving them and inspiring them by service.

Publications

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 27 Aug 2017 11:04:21 +0000

For a complete list of publications see my Google Author and ORCID profiles.

About

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 27 Aug 2017 05:16:48 +0000

About

Abhishek Tiwari is a researcher and engineering leader with deep expertise in software engineering, security and privacy, service-oriented architectures, and large-scale distributed systems. He currently serves as Editor for the Journal of Open Source Software (JOSS) and Journal of Open Research Software (JORS).

Bias and Fairness in Machine Learning

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 03 Jul 2017 14:00:00 +0000

In AI and machine learning, the future resembles the past and bias refers to prior information. There has been a growing interest in identifying the harmful biases in the machine learning. Often these harmful biases are just the reflection or amplification of human biases which algorithms learn from training data. Some training data sets such as text, medical, criminal, educational, financial etc. are more susceptible to human biases compared to others. For example, weather data is little or not impacted by human bias.

A sidecar for your service mesh

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 23 Jun 2017 14:00:00 +0000

In a recent blog post, we discussed object-inspired container design patterns in detail and the sidecar pattern was one of them. In a sidecar pattern, the functionality of the main container is extended or enhanced by a sidecar container without strong coupling between two. This pattern is particularly useful when using Kubernetes as container orchestration platform. Kubernetes uses Pods. A Pod is composed of one or more application containers. A sidecar is a utility container in the Pod and its purpose is to support the main container. It is important to note that standalone sidecar does not serve any purpose, it must be paired with one or more main containers. Generally, sidecar container is reusable and can be paired with numerous type of main containers.

What it means for a machine to learn or think like a person?

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 18 Jun 2017 14:00:00 +0000

In a recent article, Lake et al.¹ examine what it means for a machine to learn or think like a person. They argue that contemporary AI techniques are not biologically plausible hence not scalable to the extent that will enable a machine to learn or think like a person. For instance, most neural networks use some form of gradient-based (e.g., backpropagation) or Hebbian learning. Trying to build human-like intelligence from scratch using backpropagation, deep Q-learning or any stochastic gradient descent weight update rule is unfeasible regardless of how much training data are available. When comparing the human learning and the current best algorithms in AI i.e. Neural Networks – human learn from fewer data and generalize in richer and more flexible ways. Zero-shot and one-shot learnings are a hallmark of human learning and thinking. On the other end, Neural Networks are notoriously data and compute hungry.

Bloom Filters: Is element x in set S?

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 18 Apr 2017 14:00:00 +0000

The idea of Bloom Filter was conceived by Burton H. Bloom in 1970. In a nutshell, Bloom filter is a space-efficient probabilistic data structure normally used to check for set membership (i.e. Is element x in set S?). Bloom filter offers incredible memory and run-time savings for set membership queries with a small margin of error i.e. false positives. Bloom filter is used extensively in a variety of domains including implementing distributed system algorithms. There are over 20 variants of Bloom filter and these variations include additional capabilities such as counting, deletion, popularity awareness, false negative, and output type (boolean vs. frequency). Just to note, standard Bloom filter is susceptible to false positives, but not false negatives. In this article, we will discuss the basics of Bloom filter, different use cases, various considerations, important variations, lastly implement a Python/Go version of standard Bloom filter.

The Raft of the Understandability

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 08 Apr 2017 14:00:00 +0000

Computer science is full of complex hard-to-understand algorithms. Understandability of an algorithm - ability to explain an algorithm in simple terms - is generally under appreciated. Often poor understandability of an algorithm has a direct impact on practical real-world applications. In my opinion, if we can’t develop intuitions about algorithm it will be very difficult to implement or extend the algorithm for real-world use cases. There are only a few occasions when an algorithm was designed for understandability or influenced by intuition.

Object-inspired container design patterns

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 28 Mar 2017 13:00:00 +0000

If we start thinking containers in terms of objects, it abstracts away the low-level details but reveals higher-level patterns that are common to a variety of applications and algorithms. A recent paper, published by Google Researchers Brendan Burns and David Oppenheimer, suggests that containers are particularly well-suited as the fundamental “object” in distributed systems by virtue of the walls they erect at the container boundary. In many ways, this paper is seminal research publication and authors believe that containers are destined to become analogous to objects in object-oriented software systems.

Kubernetes on AWS: What you need to know and why

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 06 Feb 2017 13:00:00 +0000

Setting and running Kubernetes on Amazon Web Services (AWS) is a very involved process. AWS has decided to not implement Kubernetes as a Service but built something of its own - Amazon EC2 Container Service (ECS). Hence, you need to know and consider a lot of things before you can successfully roll out Kubernetes on AWS. These are the consideration which can make or break it your Kubernetes rollout.

Kubernetes is the right place for serverless

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 20 Jan 2017 13:00:00 +0000

Function as a Service (FaaS) remains at the core of the serverless movement. That said, most of practical serverless implementations I have seen use a combination of serverless functions and more conventional microservices running in containers. Instead of a one-size-fits-all approach, i.e. trying to use serverless function for everything, a balanced approach is highly recommended. To a large extent, Kubernetes is best placed to bring serverless functions and conventional microservices together seamlessly.

Stored Procedure as a Service (SPaaS)

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 08 Oct 2016 13:00:00 +0000

Functions as a service (FaaS) is an emerging pattern to build APIs and microservices at scale. You can use various FaaS implementations such as AWS Lambda, Azure Functions, and Google Cloud Functions to build APIs ecosystem for your organisation. If you remove serverless requirements, stored procedures can be considered as a variation of FaaS because they share similar traits and concerns. Developing APIs against these stored procedures is generally considered as an anti-pattern. But what if you have a substantial investment in stored procedures (8-12 years) and most of your business logic is embedded in stored procedures. In this blog post, I will discuss how you can leverage existing stored procedures to create a fast-track API transformation program on top of your legacy systems.

Culture eats agile for breakfast

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 23 Sep 2016 14:00:00 +0000

Implementing agile which works and delivers can be a very challenging task, especially in corporate and enterprise environment. According to 10th State of Agile Survey, two of the top five causes of failure were related to company culture – company philosophy or culture at odds with core agile values.

For next wave of innovation organisations will need internal data services

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 22 Apr 2016 14:00:00 +0000

To unlock the true value of data, organisations will need internal data services. Data services provide streamlined and centralised data access to a diverse set of users which removes the friction in delivering faster insights, products and services. Data services promote innovation. In addition, effective implementation of data services can reduce data duplication hence reducing the cost of data management and storage. Often referred as data-as-a-service (DaaS), data services are natural next step in the evolution of as-a-service model. Data services provide access to a broad range of internal and external data sources to power both internal business applications as well as client-facing software-as-a-service (SaaS) applications.

Amazon Redshift : Avoid data redistribution

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 14 Mar 2016 13:00:00 +0000

When using Amazon Redshift, distribution style plays an important role in optimising the table design for best performance. In a nutshell, table’s distribution style dictates how the data is distributed across Redshift node and slices. A key objective is to avoid the data redistribution during query execution or runtime. This is accomplished by locating or co-locating the data where it needs to be before the query is executed. For instance, if a query is performing join over two tables, to avoid the redistribution of data, data from two tables can be co-located by planning an appropriate distribution style.

The 10 Commandments of Tag Management

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 30 Jan 2016 13:00:00 +0000

Hundreds of marketing technologies, thousands of tags - effective tag management requires more than just an enterprise tag management solution. In this post, I present you my 10 commandments for effective tag management. I think these 10 simple rules are quite obvious and useful for anyone interested in implementing a more robust and effective tagging regime.

How to Organise Your Infrastructure as Code

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 22 Dec 2015 13:00:00 +0000

DevOps is a cultural shift with immediate focus on maximising the business value by opting better communication, collaboration and feedback within and across IT development and operation teams. Infrastructure as Code (IaC) is a key element of DevOps philosophy with benefits for both development and operation teams. The term infrastructure as code is sometimes also referred to as programmable infrastructure. In an infrastructure as code implementation whole infrastructure lifecycle including orchestration, provisioning, configuration, monitoring, self-healing can be managed in an automated fashion.

AWS Lambda and Endless Serverless Possibilities

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 26 Oct 2015 13:00:00 +0000

Ever since the Amazon introduced the AWS Lambda service at AWS re:Invent 2014, a variety of new applications for the service has emerged which highlights tremendous potential and traction for the AWS Lambda service. Over last one year, Amazon has been actively working towards integrating other AWS services with AWS Lambda. You can consider recently released Amazon ElasticSearch service as a good example which heavily relies on AWS Lambda to stream the data from other AWS services such as Amazon CloudWatch, Amazon S3, etc. into ElasticSearch. With recent buzz around microservices and launch of Amazon API Gateway, AWS Lambda has emerged as preferred choice to develop serverless microservices. On top of that, in recent re:Invent AWS Lambda was further expanded with new features such as versioning and scheduled jobs, and introduced support for other popular languages including Python and PHP.

Goodbye Splunk, Hello Amazon Elasticsearch Service

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 01 Oct 2015 14:00:00 +0000

Amazon today announced Amazon Elasticsearch Service (Amazon ES) - a fully managed Elasticsearch service which can support your real-time distributed search requirements. I am not sure about others, but for me a this is a big deal and can be a game changer. I have worked with both Splunk and Elasticsearch.

Reflections on Apache Drill

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 29 Sep 2015 14:00:00 +0000

I have been playing with Apache Drill for quite some time now. In layman’s terms, Apache Drill is SQL query engine which can perform queries against any type of data store - in particular any non-relational data store. Apache Drill’s ability to query against raw data files stored on the cloud storage platforms such as Amazon S3, Azure Blob Storage, Google Cloud Storage makes it a really attractive and cost effective query engine. Using Apache Drill one can get faster insights without requiring up-front schema knowledge or without loading the data in the relational data store.

Rethinking the database with drop-in replacements

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 27 Sep 2015 14:00:00 +0000

We are observing a new era in database technology space, drop-in replacements with order of magnitude improvement in performance over their counterparts. First Amazon Aurora and now Scylla DB. Although the concept of drop-in replacements is not something new but most recently these substitutes started pushing the performance boundaries using completely overhauled architecture. Apart from feature parity, the focus for drop-in replacements is shifted towards lowering operational and infrastructure costs with the ability to scale and heal.

Visualising CloudFormation Stack

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 28 Aug 2015 14:00:00 +0000

If you work with AWS CloudFormation on a day-to-day basis, I think you might be interested in visualising your AWS CloudFormation templates. CloudFormation stack visualization can be quite handy for lifecycle management of your AWS deployment. In addition, visualization can be immensely helpful in change management and governance.

DynamoDB or Aurora

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 06 Jul 2015 14:00:00 +0000

Aurora is a new offering from Amazon Relational Database Service (RDS). It is advertised as drop-in replacement for the MySQL - full compatibility with MySQL 5.6. Aurora is designed for fault-tolerance, availability and storage elasticity. It is a relational database and a highly cost effective one. With Aurora you never need to over-provision the storage, database instance volume will grow in increments of 10 GB up to a maximum of 64 TB.

Rise of Elastic Data Warehouse and Database Services

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 03 Jul 2015 14:00:00 +0000

Currently the majority of cloud based database and data warehouse services are provisioned with fixed storage and compute resources. Resizing of resources cannot be performed without compromising availability and performance. This means service users typically end up with over-provisioned under-utilised expensive resources to accommodate possible peak demand. In the worst case, under-provisioned resources unable to handle sudden work overloads. In my opinion, this is a big limitation on the part of the cloud based database and data warehouse services, especially if you consider how in the cloud one can easily auto-scale on-demand storage and compute these days.

Requirements for stream processing architecture

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 06 May 2015 14:00:00 +0000

In 2005 Stonebraker et al. published a paper that outlined 8 key requirements for stream processing architecture. These key requirements can be easily translated into building blocks of stream processing architecture. Although, this article dates before systems such as Apache Kafka, Amazon Kinesis, Apache Spark, Apache Storm, etc. most of the requirements are still relevant.

MVC for data science is here

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 02 Apr 2015 13:00:00 +0000

For those who are not familiar with MVC (Model–view–controller), MVC is a software architectural pattern primarily used for web development and to implement user interfaces. Ruby on Rails (RoR), Django, Spring MVC, ASP .net, Symfony, Zend, Express are some of the most popular MVC frameworks currently available for rapid web development. Over the time web development community has immensely benefited from these MVC frameworks. So much that a TODO list application has become something of a poster child for these MVC frameworks. In a nutshell, MVC frameworks have: 1) standardised the web development; 2) lowered the web development learning curve, and; 3) accelerated the web development and release cycle.

Multimodel Database or Polyglot Persistence

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 10 Mar 2015 13:00:00 +0000

In a recent InfoWorld article, FoundationDB’s Stephen Pimentel explained the rise of the multimodel database. A multimodel database uses a single data store to support multiple data models. For instance, FoundationDB tries to map documents, graphs, and relational tables to a key-value data store. Applications can either send their data directly to the FoundationDB key-value data store or they can speak to co-located layers representing multiple data models. Under the hood a layer is stateless and translates a data model to store them as key-value store.

Tag management: Focus on SSL- Security, Structure and Latency

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 14 Feb 2015 13:00:00 +0000

A recent white paper from Ghostery provides some interesting best practices for managing your site’s marketing cloud. Following is a rundown of key take aways from the report including my own comments.

Google Tag Manager: Don’t Repeat Yourself Conversion Tracking

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 17 Jan 2015 13:00:00 +0000

If you are working with an e-commerce website, 5 to 10 e-commerce conversion pixels are quite common these days. Ad conversion pixels, affiliate conversion pixels, web analytics conversion pixels are most common type of tags typically deployed on an e-commerce website. Most of conversion pixels are designed to collect the transaction data such as id, total revenue, tax, products etc. Depending on pixel type, data collected by conversion pixel serves different purpose. For instance, ad conversion pixels allows you to calculate cost per acquisition (CPA), whereas affiliate conversion pixels enables affiliate or third-party partners to track sales contributed by their network. Normally conversion pixel requests are triggered on the confirmation or success page once customer has completed the transaction.

The current state of ad viewability

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 06 Dec 2014 13:00:00 +0000

A recent Google report highlights poor viewability of digital ads and offers some interesting statistics around various viewability factors. Report suggests that about fifty percent of ad impressions are never seen which means basically fifty percent of display spending are just wasted. Issues around ad viewability are known for quite some time now, it was not until very recently advertisers and ad vendors started taking ad viewability very seriously. Back in 2012, 1.8 trillion display ads were paid for but not seen so things are not really improved since then. Viewability issues are more serious with non-direct inventory, i.e. display ads purchased on networks and ad exchanges. Another 2014 Q3 report suggests that two-thirds of non-direct inventory is non-viewable as per MRC Viewability standard.

How Looker is unlocking Business Intelligence using YAML

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 27 Nov 2014 13:00:00 +0000

Recently I discovered Looker, a relatively new Business Intelligence (BI) platform which utilises a YAML based modeling language LookML. LookML is core engine of Looker. LookML offers a very powerful, highly expressive yet lightweight modeling layer. LookML combines data modelling with on-demand data transformations, and data derivations at the same layer. This kind of late-binding modeling bypasses traditional ETL based data loading and warehousing.

Why software engineers will be best marketing technologist in a digital world

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 20 Oct 2014 13:00:00 +0000

Recently Clive Sirkin, the chief marketing officer of Kimberly-Clark suggested that best marketer in a digital world would be the marketing technologists. I am in complete agreement with Sirkin’s observation. Scott Brinker called marketing technologist as Neo of the marketing matrix. HBR covered a detailed story on rise of chief marketing technologist and Forbes compared marketing technologists to unicorns who will rule the modern marketing world. The role of technology in marketing is now evolving from enabler to innovate — a primary reason why marketing technologist are so much in demand. Technology now truly powers and dominates the majority of innovations in marketing.

Google Tag Manager: Visitor UUID Cookie Setter Tag

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 12 Oct 2014 13:00:00 +0000

Sometime you want to generate UUID for visitors on your web property and set UUID as cookie. I want to show how easy it is to set your own visitor UUID cookie using Google Tag Manager(GTM). Once visitor UUID cookie is set you can pass this UUID to other tags. Normally most of tags set tag specific UUIDs but by setting your own UUID you can manage your own cookie match table with your own UUID as primary key.

Jsonnet: A data templating language from Google

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 12 Oct 2014 13:00:00 +0000

Jsonnet is a new domain specific configuration language from Google which allows you to define data templates. These data templates are transformed into JSON objects using Jsonnet library or command line tool. As a language, Jsonnet is extension of JSON - a valid JSON object is always valid Jsonnet template.

Google Tag Manager: Lookup table and DRY Principle

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 10 Oct 2014 13:00:00 +0000

By default Google Tag Manager (GTM) provides several handy macros and lookup table macro is one of them. In GTM, by using lookup table macro you can reduce number of deployed tags as well as the number of rules you need to load the tags. When implementing tags in GTM always follow famous DRY principle (don’t repeat yourself). Use of lookup table avoids duplication and enables reusability. One common use case of lookup table macro is to separate Google Analytics (GA) tracking IDs for different environments - Test vs UAT vs production, etc.

Google Tag Manager: How it works

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 03 Oct 2014 14:00:00 +0000

In simple words, a Google Tag Manager (GTM) container holds other tags. GTM container is also know as universal container tag or site-wide tag. A GTM container is managed via GTM dashboard and controls tag dependencies and tag firing order. On your web page, you can have multiple GTM containers. GTM container code looks like following - embedded either header or footer of your web page template.

Google Tag Manager: Ecommerce Integration

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 26 Sep 2014 14:00:00 +0000

Believe it or not, on average an US ecommerce website has between 15-20 marketing tags. Implementing and maintaining these many marketing tags on your website is often a cumbersome job. Google Tag Manager (GTM) is a great tool for marketer as well as developers. Like any other Tag management solution (TMS), GTM not only decouples deployment of these marketing tags but also introduces an efficient way of sharing data to these tags via data layer. A best practice-driven GTM implementation can save hundred of hrs of your time. Most importantly GTM is free and equally powerful as any other enterprise tag management solution in the market.

Varnish cache buster add-on for Statamic

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 26 Sep 2014 14:00:00 +0000

Few months back I managed to get Statamic working with Varnish. Setting up Varnish for Statamic was quite easy and my website performance improved drastically (you can see load and performance testing benchmarks here). One of the biggest challenges was Varnish cache invalidation on content update. Every time, Statamic content was updated I have to invalidate cache manually using curl requests.

Unreasonable Effectiveness of Email Remarketing

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 01 Aug 2014 14:00:00 +0000

SeeWhy¹ a market leader in real-time behavioural marketing, has recently published an extensive and empirical research on ROI of 1-to-1 email remarketing programs. In nutshell, research suggests that an optimised and best-practice driven email remarketing programs can deliver significant incremental revenues (on averages 5.14% and ranged between 2%-9%). So an ecommerce business with a baseline (without remarketing) revenue of $100 million can easily generate revenue between $102 million to $109 million with remarketing. This is inline with previous SeeWhy research suggesting that businesses in the $500 million range can expect to generate up to $50 million in incremental revenues by best-practice driven email remarketing campaign.

Adobe AEM: Good, Bad and Ugly

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 27 Jul 2014 14:00:00 +0000

After working with Adobe AEM and several other JCR backed content management systems in last few years I have realised that there are a lot of mis-conceptions about Adobe AEM. As matter of fact, brain washing is quite common these days when Adobe AEM is on the table. Sometimes Adobe is on fault, on other occasions rhetoric of competitors, agencies and architects.

Digital Experience Delivery Platforms

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 26 Jul 2014 14:00:00 +0000

A recent report by Forrester Research on digital experience delivery platforms has named Adobe and Hybris as leading strong performers. Adobe was recognised for its web experience management suit which includes content management, digital assets management, targeting and personalisation capabilities. Hybris was acknowledged for the e-commerce experience management with strong presence in product information management (PIM) and digital commerce space. Apart from Adobe and Hybris, SiteCore and IBM also barely made in strong performers list.

Getting started with Vagrant

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 04 Jul 2014 14:00:00 +0000

Vagrant is a perfect choice for virtual development environment. Vagrant was created and maintained by Mitchell Hashimoto. It benefits developers, operations engineer and designers. Vagrant really shines when working in a team of any size - small, medium or large. It is great technology to avoid works on my machine syndrome and issues caused due to environmental differences between developer and production environments.

Performance testing as a first-class citizen

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 02 Jul 2014 14:00:00 +0000

Today I want to talk about the performance testing. Automated web application testing is not very rare these days. This includes functional, unit and regression tests at minimum. A good web application delivery approach normally includes a continuous integration setup to run automated tests. Every time a new change is pushed to version control system, continuous integration server should run the automated tests and depending on test results deploy latest build in a staging or QA environment.

Interacting with Tagged EC2 Instances using Fabric

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 01 Jul 2014 14:00:00 +0000

AWS allows tagging of EC2 resources using your own metadata (key-value pair). Tagging of EC2 resources is quite helpful in managing instances, images and organise billing by tag names. An EC2 instance can be tagged by CLI tools, API wrappers or using AWS Console when launching the instance as well as after launching the instance.

Marketing Middleware and Data Layer

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 29 Jun 2014 14:00:00 +0000

Term “marketing middleware” was recently pioneered by marketing technologist and blogger Scott Brinker (at least in context of marketing technology). Search for term “marketing middleware” on web leads to Scott Brinker’s posts on Marketing Technology Landscape Supergraphic and Tag management software as marketing middleware. Scott’s marketing technology landscape post is one of the most talked and referenced article this year in marketing technology space. I wrote my view on this subject with stress on the API Platforms as enabler for marketing and digital transformation.

Marketing cloud: Buy, Curate or Build

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 26 Jun 2014 14:00:00 +0000

A lot have been said and written about growing competition between major marketing cloud vendors and how they are more different than similar. We can include Adobe, Oracle, Salesforce, Google, IBM and may be Microsoft in this game of marketing cloud. With Oracle announcing unified Oracle Marketing Cloud, IBM introducing ExperienceOne, Salesforce reinventing ExactTarget Marketing Cloud, Adobe opening Marketing Cloud Exchange battle of marketing cloud is heating up.

Statamic with Varnish for a high-performance website

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 22 Jun 2014 14:00:00 +0000

So you got your website up and running using Statamic. For better performance you have enabled,

Installing Statamic

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 21 Jun 2014 14:00:00 +0000

Statamic is a new PHP CMS with relatively small community. This website currently runs on Statamic. When it comes to setup, Statamic documentation provides very brief instructions on how you can install Statamic. Normally installing Statamic is quite easy. This is a quick rundown on how you can install Statamic on Ubuntu with Apache as web server.

How to PageSpeed with Apache

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 17 Jun 2014 14:00:00 +0000

This is a quick rundown on how to install, configure and tune the PageSpeed on Ubuntu with Apache as web server.

Web Components: Web's Polymer Future

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 11 Jun 2014 14:00:00 +0000

Recently I have been playing with Web Components (WC). W3C specification drafts describe Web Components as the component model for the web ¹. This component model consists of five new constructs: HTML Templates, Decorators, Custom Elements, Shadow DOM and HTML Imports. These constructs will be used as new primitives in the browser. This includes new standard and custom reusable HTML elements. So for instance a google-analytics Web Component element,

Class and Objects in R: S3 Style

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 06 Mar 2014 13:00:00 +0000

When it comes to R Programming, I guess you can always consider me as a beginner. Although I have known basic R for quite some time now, I never tried object-oriented programming in R. Recently I started to explore some object-oriented stuff in R. To my surprise, R has three object-oriented systems: S3, S4 and R5. Creator of ggplot, Hadley Wickham maintains a brief description of all three types of system S3, S4 and R5.

Building Distributed Systems with Mesos

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 27 Feb 2014 13:00:00 +0000

Apache Mesos is a popular open source cluster manager which enables building resource-efficient distributed systems. Mesos provides efficient dynamic resources isolation and sharing across multiple distributed applications such as Hadoop, Spark, Memcache, MySQL etc on a dynamic shared pool of resources nodes. This means with Mesos you can run any distributed application which requires clustered resources.

Why are current DAM offerings so uninspiring?

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 26 Feb 2014 13:00:00 +0000

Anjali Yakkundi from Forester offers some interesting insights about current digital asset management (DAM) technologies in the market. Most of DAM vendors are either struggling to adopt new digital trends or unable to respond to new type of requirements.

Migrating to Statamic

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 24 Feb 2014 13:00:00 +0000

Migration of this blog from Jekyll to Statamic is now complete. To my surprise migration process was very smooth and quick (it took less than 5 hours). As part of migration I also ported design of existing Jekyll website to Statamic. In addition, actual content migration was quite straightforward.

Marketing Technology Landscape 2014

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 08 Jan 2014 13:00:00 +0000

Two days back Scott Brinker published Marketing Technology Landscape for 2014. This is the third edition of his widely appreciated and popular marketing technology landscapes. This version includes about 950 marketing technology companies into 43 categories across 6 major classes. These companies mainly provide software and tools for marketers. This is 600 more than year 2012 and ~9 times compared to year 2011.

Web Analytics Annotation and Application Release

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 06 Jan 2014 13:00:00 +0000

You may be wondering what can be common between web analytics annotation and application release. Before I answer that, it will be nice to understand what is annotation with respect to web analytics.

Event Tracking with Data Layer and Data Attributes

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 03 Jan 2014 13:00:00 +0000

Recently I wrote about a standard data layer for customer experience data. Extending our last discussion, in this post I am going to describe how you can use custom data attributes (data-*) and data layer for event tracking.

2014 - The Year Ahead, 2013 - The Year Lookback

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 02 Jan 2014 13:00:00 +0000

A very happy new year to all my readers. The excitement of making resolution in new year’s week is practically irresistible. So here I am. This post is for myself more than anyone else.

A Standard Data Layer for Customer Experience Data

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 25 Dec 2013 13:00:00 +0000

After much deliberation, recently W3C Customer Experience Digital Data Community Group released version 1.0 of Customer Experience Digital Data Layer (CEDDL). This specification is designed to communicate customer experience digital data to analytics and reporting services.

Templating for Responsive Images

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 24 Dec 2013 13:00:00 +0000

Recently I discussed at length about Responsive Images and Responsive Image as Service.

Responsive Image as Service (RIaS)

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 19 Dec 2013 13:00:00 +0000

A Responsive Image as Service (RIaS) offers on-the-fly responsive image generation and delivery using REST APIs or REST like image paths.

Responsive Image Distilled

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 15 Dec 2013 13:00:00 +0000

A lot has been written about current state of responsive images. Things like why we need responsive images¹², use-cases³, requirements³ and how to choose a responsive image solution ⁴⁵⁶⁷ are well documented elsewhere.

Creating a new Vagrant base box from an existing VM

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 14 Dec 2013 13:00:00 +0000

Typically my Vagrant workflow starts with a official base box provided by Ubuntu. With help of a Vagrant providervagrant up will create a virtual machine (VM) using specified base box.

Decoupling Deployment and Release- Feature Toggles

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 18 Oct 2013 13:00:00 +0000

There are key differences between deployment and release. Deployment means putting the latest stable version of code on your web application servers. Latest version of code often includes a range of features or improvements. Release means making all or selected features available to the public or end-users. In addition, one main difference between deployment and release is the ability to roll back.

Decoupling content management using content repositories

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 15 Oct 2013 13:00:00 +0000

In last 5 years or so, content management systems have evolved. New breed of content management systems (CMS) are powered by content repositories. Behind the scenes content repositories use familiar persistence managers such as relational databases, file systems, in-memory etc. That said, content repositories are more than just plain persistence storage. Content repositories offer some interesting features such as,

Measuring CDN Performance, Availability and Scalability

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 26 Sep 2013 14:00:00 +0000

When it comes to select a Content Delivery Networks (CDN) there are many factors you need to consider such as performance, availability, scalability, reliability, pricing model, features, SLA etc. In my opinion performance and pricing are key in the decision making process.

Embracing change when on peak

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 25 Sep 2013 14:00:00 +0000

In this world there are all kind of people. But in context of this post I am interested in only two type of people. On one side there are people who are not afraid of change and embrace the change in life. On the other side there are human beings who are less open to change.

Visual Regression

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 07 Aug 2013 14:00:00 +0000

A new breed of tools have started appearing on our radar supporting the very idea of automated visual regression. A visual regression testing tool performs front-end or user-interface(UI) regression testing by capturing the screenshots of web pages/UI and compare them with the original images (either historical baseline screenshots or reference images from live website). If new screenshot deviate from baseline or reference screenshot, visual regression tool will warn you. In context of web applications, visual regression tools are performing regression test on CSS changes.

CloudFront Design Patterns And Best Practices

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 31 Jan 2013 13:00:00 +0000

CloudFront is a content delivery service offered by Amazon web services(AWS). CloudFront serves static contents (images, audio, video etc) using a global network of more than 28+ edge locations. Using these edge locations, CloudFront accelerates delivery of content by serving the cached copies of the content objects from a nearest edge location. Service is highly reliable and designed to be used with Amazon S3 or any other custom origin server.

AMQP, RabbitMQ and Celery - A Visual Guide For Dummies

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 21 Jan 2013 13:00:00 +0000

Celery is an asynchronous distributed task queue. RabbitMQ is a message broker which implements the Advanced Message Queuing Protocol (AMQP). Before we describe relationship between RabbitMQ and Celery, a quick overview of AMQP will be helpful ¹².

Going Static

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 19 Jan 2013 13:00:00 +0000

Proud to say that this blog is now written in Markdown using iA Writer, generated by Jekyll static blog generator and hosted by Amazon S3 which costs me less than 1$ a month.

Reflections on Visa's High-Availability Payment Network

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sat, 19 Jan 2013 13:00:00 +0000

Way before Google-scale or Facebook-scale existed, there was Visa-scale. This COMPUTERWORLD article from 2001 provides some insights about early Visa payment processing infrastructure.

2013 - The Year Ahead, 2012 - The Year Lookback

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 11 Jan 2013 13:00:00 +0000

First of all Happy New Year to you all. Honestly speaking this post was due in first week of this month but it’s not too late for new year’s resolution. As matter of fact I desperately wanted to write this post - to share my excitement for year 2013 and also humbling experience in 2012.

Jekyll, Amazon S3, Vagrant And Puppet

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 08 Jan 2013 13:00:00 +0000

Recently I moved my blog from Scriptogram to static site generated by Jekyll and hosted on Amazon S3. I am a big fan of Vagrant and I really like the way it keeps clean separation between various development environments without affecting host operating system. This post will explain how you can set-up a Vagrant instance for Jekyll based blogging supported by S3 hosting.

My Vagrant Workflow

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 19 Oct 2012 13:00:00 +0000

For those who never heard about the Vagrant, it is a very handy way to manage, create and destroy headless virtual machine (VM) environments.

Hadoop Ecosystem- Deployment And Management

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 12 Oct 2012 13:00:00 +0000

My notes and thoughts on Hadoop Ecosystem from book Hadoop Operations¹.

Traditional Ways To Solve Scalability Problems With RDBMS

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 05 Oct 2012 14:00:00 +0000

Notes plus thoughts from my recent read Cassandra: The Definitive Guide. Common ways to solve scalability bottleneck with relational databases,

Polyglot Persistence Patterns

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Wed, 29 Aug 2012 14:00:00 +0000

Polyglot persistence which takes a hybrid approach to persistence is getting a lot of traction these days. Currently I am reading NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence by Pramod J. Sadalage and Martin Fowler. Basic idea behind polyglot persistence is to use specialized databases (both NoSQL and Relational) for different purposes within the same web application. One-database-fits-all is no more the de facto choice for web applications. In fact polyglot persistence is about choice - choice to select best data store for a given data type or purpose. Polyglot persistence leverages the strength of multiple data stores.

Blog Reloaded

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 24 Aug 2012 14:00:00 +0000

This blog is reloaded- in terms of theme, content and target audience. For regular readers and visitors of this site/blog, this is a drastic change. If you came here looking for a bioinformatics and system biology blog then you will be fairly disappointed.

Mapreduce and Hadoop Algorithms in Bioinformatics Papers

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Thu, 09 Aug 2012 14:00:00 +0000

Solely inspired by Atbrox’s list of academic papers for Mapreduce & Hadoop Algorithms. Unlike computer science where applications of Mapreduce/Hadoop are very much diversified, most of published implementations in bioinformatics are still focused on the analysis and/or assembly of biological sequences. As usual this list will be updated time to time. If you find that any important paper that is missing from the list then please drop a comment at end of the post.

Scala for the Python Geeks

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 16 Sep 2011 14:00:00 +0000

Scala is a JVM languages designed by Martin Odersky in early 2001 and it brings features of object-oriented programming and functional programming together. Scala is competing again several new JVM languages including but not limited to Groovy, Clojure, JRuby and Jython.

Periodic Tasks and Queue Processing in Django

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Fri, 02 Sep 2011 14:00:00 +0000

In Python/Django Celery with RabbitMQ is widely used for background processing or distributed task queue. Although Celery is really focused on being a distributed task queue, it can also be used as scheduler using it’s periodic tasks feature celerybeat which kicks off tasks at regular intervals.

Knowing your data vs relying on it

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 31 Oct 2010 13:00:00 +0000

It is good to know your data. But there is clear distinction between being data driven vs data informed. No matter which area you work, there is always an opportunity to make additional gains by closely observing the characteristic and quality of your data. By experimenting and looking carefully at the data you may identify some hidden patterns which may not be visible or obvious. But can we rely on peculiarities of the data? My answer is BIG NO. For instance, recently Opera claimed that their product Opera Mini saves users worldwide more than 2.2 billion USD per month or more than 27.4 billion USD per year. I think this was ridiculously incongruous and unreasonable claim just made up by playing with numbers. According to their claim,

Bloom filters for bioinformatics

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 21 Jun 2010 14:00:00 +0000

The Bloom filter was originally developed by Burton H. Bloom back in the seventies and for long time it was there without any major application. Google is credited for making Bloom filter popular again. Only after the Google used Bloom filters for their BigTable database system, the idea started grabbing the attention of larger and diverse audience.

Put some breathe life in your papers with clever visualization

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Tue, 02 Mar 2010 13:00:00 +0000

I often screen papers for my reading list based on their illustration appeal. I know this may be bit strange for the people who judge the papers simply based on their abstracts or conclusion section. But trust me it works because I know what I am looking for. I always avoid the sleep inducing papers full of creepy tables and bar graphs, in fact so lacking in interest as to cause mental weariness.

Visual analytics: Aimless wandering or Systematic data exploration

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Mon, 23 Feb 2009 13:00:00 +0000

With great power comes great responsibility.

One of my favorite quote from movie Spider-Man which was sourced by Stan Lee, famous American writer and editor. After working with visual analytics for three years I realized that this quote is utterly correct for visual analytics. Visual analytics is a new exploratory paradigm which combines visualization, human factors and data analysis. Human factors such as interaction, cognition, perception play central role in the final outcome of the analytics and decision making process.

Circular/Radial Visualizations in Bioinformatics

abhishek@abhishek-tiwari.com (Abhishek Tiwari) — Sun, 08 Feb 2009 13:00:00 +0000

Information visualization is in the core of bioinformatics, and over the time variety of interesting information visualization approaches have been developed to visualize the different type of biological information. This post is specifically targeted on circular/radial visualization in biological domain.