Beware data scraping

Don’t we all hate bots? Sometimes they are just an irritation – but in the context of a Port Community System (PCS) or other platform providing real-time or near-real-time information exchange vital to smooth cargo flows through ports, data scraping bots could bring things to a virtual standstill.

Parvez Mansuri, CEO, Advent eModal, says the practice of data scraping is causing increasing problems for genuine users of PCSs and other logistics information portals. But what exactly is data scraping and why does it matter?

In short, scraping is when a computer program can extract data from output generated from another program – mostly, it applies to web scraping, used to extract valuable information from a website.

“A lot of our solutions depend on fluid, efficient data flow between us and others’ systems, to facilitate the movement of the container across multiple stakeholders and avoid friction,” says Mansuri. “However, there is a mis-use of the data, scraping being one of them, that impacts our traditional partners who are sharing the data through a normalised channel. Indirectly we get impacted by unfair scraping practices by various stakeholders, and this increases the burden on our customers.”

Scraping of reasonably static data – for example, a fashion website, for use in a fashion catalogue – will likely not make a big impact, he says. But in the ports and logistics context, the data is changing all the time. “This means they have to keep continuously scraping, i.e. reloading that page every second. The result can be that humans/normal users trying to gather information in a rightful way always have latency on screens, slowness on the web – and sometimes that information becomes unavailable.”

THE TICKETS NEVER RUN OUT
There are parallels with a ticketing website for an in-demand festival or concert being jammed by customers – but with one difference. “In our context, the tickets don’t run out. It never ends, it is continuously ongoing.”

Imagine a PCS or other portal has been set up with the knowledge that it will be serving 10,000 truckers, each with ten despatchers, so 100,000 potential users. The system is designed and resourced to ensure an efficient and seamless experience on the website for those users. “Then suddenly one company is pulling your website every second – and instead of 10,000 you have a 10 million kind of consumption.”

The use of data scraping bots reflects a desire for ever more visibility, Mansuri notes. For example, a despatcher at a trucking company operating in a major port might be going to the websites of a dozen different terminals. “To make that efficient, truckers started deploying this scraping technology that was in use in other markets. It’s a very simplistic technology – but it is also a blunt tool for a significantly sophisticated problem.”

It’s easy for truckers or logistics providers that don’t have enough IT resources to outsource to scraping tech companies, he says. “Third party providers make it very easy and very cheap. It works 50-60 per cent of the time. But that is another part of the problem – consumers subscribe to multiple scraping sources so they can validate that the information is accurate. This multiplies the impact.”

None of this is illegal per se – but it could be if the website’s terms of use specifically and explicitly prohibit scraping activities.

Many perfectly respectable entities will say they have a ‘visibility company’ providing visibility data but do not understand how this data is being collected, he says. Also, the information obtained in this way provides the basics (container unloaded, container in the yard) but not the detail (is the yard open or closed for delivery, can you pick up the container?).

What is the solution? “Terminals should have a secured website that requires users to identify themselves and accept terms of use. Put it behind a pay wall or security wall.”

Opting for APIs will provide a more secure approach, he adds. It’s also possible to put specific containers on a ‘watch list’ – so it’s clear who is watching each container, users can be tracked and there is no anonymous use. Advent eModal does get impacted by scraping but “we have good mechanisms to track who is doing this”.

Often, for the websites affected, this is a case of legacy practices and not understanding or quantifying the impacts – which can be considerable, says Mansuri. “Data scraping on your website can make its way into the wider IT system and slow everything. It depends on how networks are designed – some separate critical and non-critical infrastructure, but some don’t. In the latter case, it means that every time you have a scraper attacking your website, it’s possible your TOS is slowing down because of it.”