Our large scale production capabilities

We're often asked, as a small business, if we have the ability to handle massive amounts of records. Specifically, what is our background and track record in this area? Well, in terms of just scale, Qbase personnel have tackled some of the largest search challenges out there, including a system comprised of over 10,000 distinct sources of structured data collections with over 8 billion records. In addition to tackling enormous collections of data, Qbase has designed, built, and deployed flexible solutions that provide a balance between high-precision and wide-range recall. These solutions have search response times of less than one second and support over 100 queries per second. Our ability to deliver these solutions is due in large part to our team members' unique skills in data preparation and design, as well as in high throughput computing, data compression, search algorithms and system architecture. See the difference a talented team with years of large-scale production experience can make on your next project.

Unique problems of diverse data sets

If you are in the search or data business, or collect this data for yourself, you know firsthand about the search and data challenges found in diverse data sets from military, healthcare, law enforcement, research, insurance, and financial domains, to name just a few. But data from each of these different domains present a unique set of challenges because, although they may share some characteristics, they are actually quite different when viewed in depth. Qbase excels at processing legal data, for example, because our personnel have a deep understanding of the citation and precedence system interwoven into case law documents. We successfully process news data, on the other hand, because our technicians have set up systems to handle the constant stream of new data and built the necessary automated topic classification and entity extraction systems. With public records data, Qbase technicians understand know how to process the billions of highly-interrelated information fragments because we have a true understanding of the data, its characteristics, and the advanced algorithms. And our team fully understands that public records (such as healthcare data) bring with them issues involved in privacy protection and legislative compliance, and we are prepared to deal with those issues. Qbase brings years of in-depth experience across multiple data domains when solving problems in diverse data sets. Let us help you solve your unique data problems.

How Qbase analyzes data

When we start data projects for clients, we use only our Qbase organic tools. That way, we don't need to rely on licensing or installing third-party software, or bringing anyone up to speed on external technologies — we just get to work and get results. We start by using our data discovery tool, Qbase Data Discovery™ (QDD™). QDD™ identifies the type of data in a field automatically, based on dynamically-created definitions for a particular data domain. QDD™ not only profiles the data, but QDD™ also exports the data and associated metadata for use in downstream processes. As part of the valid data definition process, we optionally present results of the discovery to the client to help identify data issues in real-time. Our tools and methods give Qbase a true advantage over our competitors who use slower and more expensive technologies. In other words, Qbase’s tools present real results faster, so you can start making inroads on data migration projects sooner, with less overhead, than any other system or supplier out there. Learn more about Qbase technologies at http://www.qbase.us/our_technology.html

More about data fusion...

In our last post, we were discussing how Qbase is able to handle data fusion in realtime. The secret is in how we use RAM (Random Access Memory). The indexes we create are highly compressed, but contain all required key fields. The in-memory indexes can then be distributed across a huge number of large memory servers. Scoring algorithms are all absolute, based on a ‘0-to-1’ continuum of values which allow queries to be efficiently fanned out to hundreds of servers and then collated back into a single ranked answer set. Qbase personnel have developed systems where data from thousands of sources were fused into a collection of approximately eight billion records, all stored in RAM. Response time averaged less than 0.25 seconds for core queries. Data was distributed across 150 large memory machines totaling nearly five terabytes of physical memory. In fact, Qbase projects involving geospatial imagery currently process and store over 200 terabytes of data and support expansion to 12 petabytes. Improve your decision response time and results with Qbase.

What is Qbase's approach to data fusion?

We've had some questions recently on what the Qbase approach is to fusing disparate data sets together. Well, unlike some of our competitors, at Qbase our unique approach to data fusion is based completely on runtime matching and linking, which means there is no need for linking of any data in advance. What our technicians have done instead is index the data based on “loose keys.” The resulting queries bring back relatively large answer sets that are then scored, ranked, and rolled up into linked collections. There are a couple advantages to doing it this way. First, new data can be added to the system as soon as it’s available which can immediately participate in linked answer sets — there are no complicated linking processes to hinder data from being available as soon as possible. Second, run-time linking provides the ability to dial-in varying link strengths or to try different link algorithms without re-fabricating the entire data collection. Give us a call to discuss your challenges — we’ll convert them to successes.