Security teams are increasingly frustrated with legacy solutions that are not designed to address the data volumes they face today. Threat hunting and incident investigations are hindered by searches that take too long to run or simply time out. If the searches do finally run, we quickly discover critical data is missing because it was deliberately excluded due to either the costs associated with indexing it or the long term storage costs, ultimately yielding an incomplete picture.
To begin solving the problem, Hadoop becomes the first logical choice. It’s free, it is scalable, and it’s fast. Once Hadoop is in place, an interesting shift starts to occur inside the SOC – suddenly there is new demand to get more data in, extend access to more users, and of course the data has to be kept safe and secure. However, these are actually all good problems to have, you just need the right approach to ensure that Hadoop never becomes just another isolated data silo with limited data.
At JASK, we believe Hadoop is the new core of the SOC for the same fundamental reasons. We leverage Clouderas Hadoop distribution to enable security teams to capture, store and process, and drive value from security data – at massive scale – all while providing everything your organization needs to keep data safe and secure while still meeting compliance requirements.