skip to Main Content

Querying Data Streams in Real-Time

Processing and acting on high volume real-time data is a requirement for many organisations today including:

  • Financial: Processing real time trading information
  • Utilities: Monitoring power consumption, tracking supply vs. demand
  • Communications: Monitoring network usage, tracking failure rates, Processing Banner Ad clicks on web sites

However the traditional approach to data analysis is; to store the data in a database, run queries to extract and calculate the required data and then to produce output in the form of a report. But with 1000 new data records arriving per second this was clearly not practical, until StreamInsight came along that is.

StreamInsight is part of SQL Server 2008 R2 and is designed to accept and query real-time data streams with volumes of thousands of records per second and it is all based on familiar .NET and LINQ technology. There are two editions:

  • Standard edition for data volumes less that 5000 events per second
  • Premium edition for data volumes up to 100000 events per second

You need SQL Server 2008 R2 Data Centre Edition to get the Premium edition, while Standard edition comes with SQL Server 2008 R2 Standard and Enterprise Editions. So how does it work?

  • Firstly the data stream is read from the sources by an adaptor. StreamInsight comes with a set of standard Adaptors for typical data sources, but new custom ones are typical created from these
  • The data is then passed to the queries, which are written using the LINQ query language. These queries operate on the data as it arrives
  • The results of the queries are sent as a data stream to output adaptors which in turn route the results to the system that needs them. This may be a reporting solution or a trading application and so on

Architecturally it looks like this:

Querying Data Streams In Real Time

StreamInsight typically runs as a Windows Service on a server, but can be embedded directly into a program if required.

All in all this is a pretty cool extension to SQL Server and real-time decision making on high volume data can now become a part of any organisations analysis and reporting capabilities.

By Geoff Ballard, Chief Technical Officer

Have a comment? Let us know below, or send an email to [email protected]

About the Author

Geoff Ballard

As Co-Founder of Ballard Chalmers, Geoff is the company’s CTO, directing technical strategy, overseeing technical consultants, managing larger development projects and ensuring technical delivery quality standards.

Education, Membership & Awards

Geoff graduated from the University of London with a BSc (Hons) in Mathematics and computing and a Masters degree in Database Management Systems. Additionally, he is a certified SQL Server Development, Microsoft Certified IT Professional (MCITP) and Microsoft Certified Technical Specialist (MCTS).

His professional memberships include Charted Engineer (Software), Member of the British Computer Society (MBCS), Member of European Federation of Engineers (FEANI) and Fellow of the Institute of Analysts and Programmers (FIAP).

 

Our Technologies

HAVE YOU SEEN OUR NEWSLETTER?

ABOUT US

Ballard Chalmers is one of the UK’s leading software development companies, specialising in engineering enterprise applications on the Microsoft Platform. We are principally dedicated to custom software development and integration for the Microsoft Cloud, as well as on-premises and hybrid systems. Our expertise is in Microsoft .NET and Azure, SQL Server, BizTalk Server, SharePoint Server and Xamarin.

01342 410223

[email protected]

 

CONTACT US






This form collects your name and details so that we may answer you. Check out our Privacy Policy for full details on how we protect and manage your data.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

© 2020 Ballard Chalmers Ltd. All Rights Reserved

Back To Top