The rise of lightweight BI

BI augmented with AI

S&P 500 by columns.ai

Today, we are going to delve into an exciting topic that is gaining momentum in the world of data management and business intelligence — lightweight Business Intelligence, or lightweight BI. Lightweight BI is a modern variation of the long-lived term Business Intelligence (BI). This emerging trend is redefining how individuals and organizations handle data, making it a crucial topic that matters to all of us.

Before we dive deeper into today’s discussion, it’s important that we ask ourselves a few key questions. These questions will not only provide us with a better understanding of the subject matter but will also help us identify our personal experiences and challenges in data management and business intelligence.

1. Have you ever encountered challenges when attempting to manage and make sense of your data?
2. Have you been facing complications or difficulties in using your current BI tools effectively?
3. Have you experienced any obstacles or issues in your interactions and collaborations with your team members, particularly when it comes to making data-driven decisions?

By reflecting on these questions, we can better appreciate the significance of lightweight BI and understand the potential impact it could have on our work and overall business operations.

The Importance of BI

In today’s digital-driven world, nearly everything is processed or recorded as data in some format through devices such as phones, cameras, sensors, POS systems, and registration tablets.

Our world is adorned with and driven by data. Business Intelligence (BI) helps us construct a comprehensive understanding of our business and the world around us through this data. BI is the tool that allows us to make sense of it all.

Traditional BI

There are numerous BI tools available in the market, with Tableau and Power BI being the most notable. With tenures spanning over a decade or two, these tools are versatile in constructing BI pipelines, mainly focusing on two demanding tasks:

• Data modeling
• Data visualization

Data modeling is a complex process. It typically requires a profound understanding of the network of data tables or data sources required to establish a comprehensive business view.

Joining multiple tables, using various stages of data preprocessors and post-processors, and writing transformation functions to achieve the desired data model is commonplace.

Likewise, it’s normal for data model updates to be ongoing as new requirements from individuals are collected, or in response to changes in business logic. This is why businesses or organizations employ full-time “BI” engineers or scientists.

The Problems

There are several challenges, which may not seem like real problems unless viewed from a different perspective:

1. The “data modeling” process is burdensome and not agile enough to quickly adapt to business changes.
2. The existing data model pipelines are fragile and can break easily if any underlying data structure changes. This results in high maintenance costs.
3. BI data modeling/pipeline is like storing your data twice in a BI tool, leading to extra costs for support and operation.

For large organizations, managing a BI team can be challenging, yet it’s often necessary due to the increasing importance of BI.

Modern Data Stack & Lightweight BI

You’ve probably heard of the “modern data stack”. Its core concept is the adoption of cloud data warehouses. Infrastructure providers like Snowflake, BigQuery, and Databricks offer cost-efficient data warehouse products.

As users, we worry less about data availability and computing resources, paying only for what we use. With competition, data storage and computation become affordable with reasonable response time.

This context encourages us to reconsider BI and embrace lightweight BI. But, what is lightweight BI?

Lightweight BI, as the name suggests, avoids the costly heavy lifting of traditional BI and focuses on the most valuable aspects:

interpreting your data and telling its story.

Lightweight BI doesn’t natively support data modeling. Instead, it expects your “Data Model” to be constructed quickly at runtime due to the scalability of cloud computing (e.g., Snowflake’s elastic computing strategy). This means you don’t build data models beforehand, but at runtime.

Your data model might already exist as a wide table or a flat file: Spreadsheets, CSV, a SQL query, Notion database, or Airtable, for example!

Lightweight BI supports exploration of all dimensions and metrics from the data model. Whether you’re doing aggregations or runtime data transformations, it can be done in sub-seconds, offering responsive UI operations.

Moreover, lightweight BI emphasizes data storytelling and communication, which are critical for non-technical leaders. The true value of BI lies in communicating insights.

The Role of AI

In the modern world, Artificial Intelligence (AI) plays an instrumental role in aiding the storytelling process. This is achieved through several methods:

• AI is capable of generating a plethora of new ideas. This is particularly useful when there are potential insights hidden deep within the data that may otherwise go unnoticed.
• The technology can construct narratives based on connections found in the data, whether this be in the style of a document or in a presentation format.
• AI is also a valuable tool when delving into a specific story, guiding the user through the storytelling process to ensure maximum impact and understanding.

On the other hand, lightweight Business Intelligence (BI) is a powerful tool that offers immense value to its users. It achieves this by:

• Completely eliminating the need for an additional data modeling pipeline, thus streamlining the process and increasing efficiency.
• Assisting with data analysis, exploration, and transformation at runtime, which allows for a more flexible and adaptable approach.
• Effectively managing data sources between various roles within an organization, ensuring smooth communication and a unified approach.
• Prioritizing storytelling and communication, thus making it easier to understand complex data and share insights with others.
• Being driven by the assistance of AI, which enhances its capabilities and allows for more advanced, sophisticated usage.

In conclusion, both AI and lightweight BI serve as valuable aids in data analysis and storytelling, each offering unique benefits and contributing to a more efficient and effective process.

Future

I wholeheartedly advocate for lightweight BI. Born from the cradle of modern technological advancements, it’s destined to thrive, thanks to its unwavering focus on the core pillars of BI — storytelling and communication.

Are you wrestling with outdated, cumbersome BI architecture and tools? Don’t just endure. Rethink. Give lightweight BI a chance in your organization. It’s not just a choice; it’s an investment in a more prosperous future.

Your team doesn’t just need a lightweight BI — you absolutely deserve it!