Here I’d like to share my experience and pain in building L7 data protection solutions which are frequently called WAF/ngWAFs or RASPs. I started to build it back in 2009 from a simple detection logic based on self-adopted heuristics for a CTF competition and then build an entire company on machine learning defined grammars. In the CTF times, I understood in practice that it’s not only necessary to build proper detection logic but also to implement it properly. I should also mention that I bypassed mod_security, Imperva, Incapsula, F5, Barracuda, Citrix and other WAFs frequently during various public competitions and bug bounty programs, therefore this article could be useful for both building and testing/bypassing purposes.
#1 Deep Data Flow Analysis
Before we can apply any detection logic, like regexps, heuristics, statistical approach or black magic, we should first decode the request. The funny thing is, there are technically multiple levels under the hood of L7+ in the modern web. In times of CGI it was possible to apply the logic right to the GET/POST/COOKIEs parameters, but over the last 10 years it became necessary to decode data from JSON/XML/Base64 and so on because each particular web application implements a custom data protocol under L7 and requires the WAF/RASP to support it completely to ensure protection.
Moreover, a lot of web apps are using multiple nested encodings like Base64 inside XML or JSON which is almost a de facto standard for enterprise apps nowadays. This addressed WAF/ngWAF/RASP vendors to the implementation of deep data flow analysis. In legacy solutions, this task can only be solved by manually configuring the structure of these data flows, which is not a feasible approach for complex enterprise applications. Modern vendors suggest semi-automatic or automatic decoding of data flows but almost nobody is doing it as deeply as needed.
#2 Detection Logic/Signatures
Detection solution should detect something, right? A lot of people believe that it’s possible to prevent attacks by regular expressions or other kinds of signature-based methods like fingerprints for tokenizers. The reality of these methods is that it is possible to prevent only a particular attack sample but not an attack type in general. It’s still so easy to apply simple obfuscation techniques or find an unsupported context for tokenizer. Some of the examples are in my previous posts (https://medium.com/@d0znpp/how-to-bypass-libinjection-in-many-waf-ngwaf-1e2513453c0f and https://twitter.com/d0znpp/status/864473897223282688).
In few words, regular expressions cover only regular grammars (by Chomsky hierarchy), tokenizers cover only context-sensitive grammars, but the real SQL/HTML/JS grammars are recursively defined. If you would like to read all the details, just download this whitepaper: https://wallarm.com/resources/evolution_of_detection/ (email required).
In the case of RASP solutions, all the same things apply. The difference is only in where the data flow will be intercepted, at the network layer or in a function call inside the app. RASP can intercept a mysql_query() call with all the arguments which will be strings. It’s even possible for some RASPs in some cases to trace the data flow to make sure that the query argument was modified by user data from the request. But what’s next? The value of this argument is still a string and still should be checked by some regexps/tokenizers. Try to imagine that it’s not a well-known MySQL but Cassandra, Redis or smth like this. This is a limitation that almost all the RASPs vendors are not keen on talking about.
#3 Security Rules Adjustments
Nobody wants to receive useless alerts. In an ideal world, we want to be just protected, not flooded with alerts. Basically, we used to evaluate the detection quality by false positives rate, but how can you formally determine what a false positive is?
In the intrusion prevention world (and all the RASPs/WAFs are host-based or network-based IDS/IPS systems of course), an attempt to do something bad called “attack”. “Something bad” means exactly that attacks targeted to identify if your system has some vulnerabilities which they want to exploit or not. Sometimes your protection system will make a mistake and mark some legitimate user traffic as an attack. This situation is usually called false positive.
But what if the attack is really dangerous but not for exactly your application? Is it a false positive or not? For example, if I copy an attack which hacked my WordPress blog and send to my friend through Facebook Messenger, should messenger.com block this message as malicious attack or not? Technically it is exactly the same attack but applied to a different website. Let’s call this as a “second-order false positive”. And that’s why all the rules should be adjusted even if no classic (first order) false positives are being caught in your detection logic.
To pinpoint that the particular application endpoint should normally have some attacks in a data flow, the detection system should know about vulnerabilities inside the application.
The second important point here is the CI/CD approach. Companies nowadays want to deploy code to production frequently, in some cases several times per day. It means that security rules should be adjusted at the same rate which is really hard with manual tuning. This problem can be solved by integration with test suites and extensive training of the rulebase during the integration with good coverage of the tests. In my practice, it was impossible to ask customers to generate tests with all the negative scenarios to train detection system that’s why machine learning with reinforcement by autogenerated fuzzing tests was implemented in Wallarm.
The world of web applications is not CGI-based anymore, we have RESTfull APIs, WebSockets, Single-page apps and sophisticated data flows inside endpoints. To avoid generating high rates false positives and to prevent easy bypass techniques, the modern L7 intrusion prevention systems like RASPs, WAFs, ngWAFs should implement something more intelligent than just a signatures or tokenizers and have a way to build application business logic behavior with deep data flow analysis.