Feature: The bot attacks bypassing e-retail defences

Andy Mills, VP of EMEA for Cequence Security talks about e-retail protection.

Automated attacks continue to plague e-retailers, with the 2024 Bad Bots Report revealing the sector saw the highest volume of bot attacks last year. Bots enable attackers to mask their activities by generating seemingly legitimate traffic while they exploit the features of web apps and mobile apps. Typical actions include stock buying whereby bots are used to price check and snap up merchandise in seconds or content scraping to learn how a system functions. These can have severe impacts, from loss of intellectual property, to increased IT infrastructure and banking costs and lost customers.

Consequently, retailers have sunk significant investment into bot detection and mitigation solutions that use website analytics to track activity and to determine if the traffic on a website is a bot or not. But, as the statistics reveal, attacks are still getting through. It’s a situation that continues to mystify and frustrate retailers. More often than not, the reason attacks are continuing to hit is due to an over emphasis on analysing web traffic. This effectively means that while the front door is being guarded the back door is open, leading to the abuse of Application Programming Interfaces (APIs).

Web and mobile apps are increasingly built on a framework of APIs as developers seek to quickly and consistently roll out services. Attackers will then probe the network for these public-facing APIs and conduct reconnaissance by sending requests to detect the make-up, functionality and access the API has to other APIs or back-office data. In fact, API endpoints provide bot overseers with such a convenient target that the bad bot report found 30% of all API attacks are bot-driven and the chances are those attacks will go completely undetected if they are carried out against shadow APIs.

You can’t protect what you don’t know is there

Unknown, unmanaged and unprotected, shadow APIs are to all intents and purposes invisible to the business and are a top target, as revealed by the API Protection Report: Shadow APIs and API Abuse Explode that found that out of the 16.7 billion malicious requests that were observed, close to a third (5 billion) were against shadow APIs. In addition, APIs that are not updated or deprecated or lack sufficient authentication also pose a risk, with recent research suggesting that as many as one in ten are vulnerable to attack due to these issues.

Once the API has given the threat actor control over a compromised application, account or multiple accounts, they can commit fraud using customers’ payment details, gift cards, or by exploiting multiple accounts to buy and resell high-demand merchandise. These Account Takeover (ATO) attacks are incrementally growing year-on-year at a rate of 10%, according to the Bad Bots report, and much of that growth is due to APIs, with 44% of ATO attacks carried out via these endpoints compared to 35% in 2022.

ATO is a problem that more and more organisations are facing as threat actors look to steal gift cards, access one-click purchasing and dominate hype-sales through buying and reselling the inventory. It may involve the use of brute force, flooding a web app with traffic, or credential stuffing whereby thousands of logins attempts are made in a short timeframe. Or it may see the attack take a more low-level approach, using compromised usernames and passwords thereby simulating genuine login behaviour. This means that ATO does not necessarily see the mass bombardment the systems and these low and slow attacks can be difficult to detect and defend against.

These detection issues reveal that bot defence alone cannot mitigate the threat posed by automated attacks – it must be combined with API protection. The link between bots and APIs has been proven again and again and we’ve now seen the OWASP API Security Project Top 10 threats amended to reflect this. Last year it published a new list with API6:2023 – Unrestricted Access to Sensitive Business Flows added to allow for the harm that can occur when API functionality is used excessively and in an automated manner. All of the examples cited refer to e-commerce, from scalping a site before the launch of a new games console, to booking 90% of seats on a flight via an airliner’s website and then cancelling them to force them to be released at a discounted rate, to abusing a referral/loyalty program to secure free rides on a ride sharing app.

Working in harmony: how bot mitigation and API defence work together

So how does a combined approach to bot and API security work? To start with, there’s a need to discover any uncharted APIs and to then monitor the API footprint for any unusual call requests. This can only be achieved by crawling the network to look for shadow or zombie APIs that have not been deprecated correctly to build an accurate inventory of the API footprint. The discovery process also presents an opportunity to take an ‘outside in’ perspective, enabling the business to see its estate from a hacker’s perspective and where the weaknesses may lie. APIs can then be classed according to the level of risk they represent to the business in terms of their role and access to other systems and data.

Knowing how bot attacks unfold is also important. Automated attacks against APIs will typically pivot to advance the attack and to evade detection. This makes behaviour-based threat detection with machine learning essential as it is able to both track and anticipate the possible ways in which the attacker will choose to escalate the attack. It does this by comparing the activity being observed against a database of behavioural patterns or fingerprints which have been created by analysing API headers and payloads and the behaviour and intent of attacks.

When it comes to mitigating the attack, options include blocking, rate limiting, geo-fencing, logging, and deception or honey-trapping, a technique that misleads the attacker into believing that the attack has been successful. This is particularly useful as a strategy as it reduces the likelihood of the attacker pivoting multiple times which then makes it easier to track the attack. Attackers also do not have limitless time and resources and if they cannot achieve their end goal within these constraints it is likely they will simply abandon the attack and seek an easier target.

Of course, it’s also important to consider the user experience too. All too often, bot mitigation sees impediments introduced that create friction in the customer journey. A good example here is the dreaded CAPTCHA that provides the user with a warped password or code or a set of images to identify. Not only do these prevent customers from progressing but they can go wrong, seeing the user repeatedly prompted. In fact, research from the University of California, Irvine suggests AI is more successful at solving them than humans who tend to only solve them 50-84% of the time compared to a near perfect score by the computer.

A way of solving this is the proof-of-work CAPTCHA which sends a request to the user’s web browser to solve a cryptographic work item to complete. The user is completely unaware of this as it happens behind the scenes, authenticating their session on their behalf. As malicious bots tend to have a difficult time masquerading convincingly as a web browser and can’t complete the proof-of-work CAPTCHA without being modified to do so, they’re unable to pass the test. Even if the bot were modified, it would suffer a second or so penalty for each initial request, slowing down the attack to the point where it no longer comes feasible. So, incorporating a proof of work CAPTCHA into bot and API defence can make a real difference.

Looking to the future, it’s a given that we can expect both good and bad bots to become the dominant form of web traffic, overtaking their human counterparts. It’s a shift that will be accelerated by AI which means we can also expect AI orchestrated bot attacks. These will pivot much more rapidly in response to rate limiting and blocking, making deflection and deception the more effective form of defence. E-retailers need to be ready for this new era by putting in place API discovery and improving their ability to detect and mitigate automated attacks and vulnerability exploits. By adopting a dual approach that combines API protection and bot mitigation measures today, they have the opportunity to get ahead of that curve.