Defining Fast Data

Big data is often created by data that is generated at incredible speeds, such as click-stream data, financial ticker data, log aggregation, or sensor data. Often these events occur thousands to tens of thousands of times per second. No wonder this type of data is commonly referred to as a fire-hose.

Fast Data is real-time big data not yet stored. Fast Data processing sits in front of the big data fire hose, sifting through the massive amounts of incoming information to identify actionable business opportunities or threats. While this data is in motion, it offers opportunities to respond immediately based on insights derived from deep, real-time analytics.

The benefits of big data are lost if fresh, fast-moving data from the fire-hose is dumped into HDFS, an analytic RDBMS, or even flat files, because the ability to act or alert right now, as things are happening, is lost. The fire hose represents active data, immediate status, or data with ongoing purpose. The data warehouse, by contrast, is a way of looking though historical data to understand the past and predict the future.

* Source: TIBCO - Get the Fast Data Blueprint

Technology that makes it possible

Fast data doesn’t refer to a particular type of data or technology. Instead, it’s data moving at very high speeds. What’s innovative is new big data technologies that can capture this data in motion and analyze it to address top business challenges like:

Internet of Things (IoT): Optimize availability, performance, capacity and resource utilization
Enhanced security intelligence: Predict, prevent and act on security threats and real-time fraud detection; increase situational awareness
Next best action: Act on up-to-the-second observations, while the event or transaction is still happening
Real-time sentiment analysis of social media: Effectively respond to improve the client experiences

There are different kinds of technology available to analyze fast data including in memory databases, complex event processing or operational intelligence solutions. But there is only one technology (stream computing) that achieves the best performance and is the broadest adopting according to research.

Fast Data Analytical Products

Even as we write, there are new products spawning to address the growing need of Fast Data for advanced streamed analytics. Some of the popular products/frameworks making Fast Data possible are:

In-Memory Computing

TIBCO ActiveSpaces

Apache Spark (Framework)

Complex Event Processing

TIBCO BusinessEvents

Data Streaming

TIBCO StreamBase

Spark Streaming (Framework)

Visualization Analytics

TIBCO Spotfire

TIBCO JasperSoft

Apache Framework (Zeppelin)