Architecture introduction

The first consideration we found ourselves making was that the platform needed to manipulate a significant amount of data and that we did not have a sufficient budget to work with standard solutions, mainly due to the cost of the hardware. We had to invent something cheap, reliable and high-performance. Obviously the cost-effectiveness constraint was the most stringent. When you have a budget everything is easier.

Some key words were immediately decided that would be the basis of all choices: scalable, asynchronous, lean. We had to avoid components that were too complex to design, write and maintain. Instead, we needed to create an ecosystem of components that communicated with each other to create something more complex. The components had to be able to run on our systems or on external systems and had to be as stupid as possible. Little logic on board and instructed by the center.

Within the platform we identified different types of components: data producers, processors, aggregators, consumers. Each component could be part of one or more categories.
The development began with the first data producers.

The first was Zefiro, the component that creates the list of recently registered domains. Zefiro uses zone files to identify new domains (and those that are deleted).

After Zefiro came Scirocco, which did similar work to Zefiro but using the Certificate Transparency Log. From here we could extract the new domains, in some cases before Zefiro found them. Obviously Zefiro had a more global vision. Using the two ensured good coverage. The component that extracts recently registered domains from the CTL feed is called Libeccio. To improve it further, Levante was born.

Levante uses OSINT channels to collect domains and paths which in some cases are not intercepted by Zefiro and Scirocco.

The last member of the data producer category was Miniluv. Miniluv processes data from the other three sources and filters and prioritizes it.

These components handle approximately 20GB per hour. If they were videos it would be a small thing, but they are text strings a few dozen characters long. Lots of stuff.

In addition to products we have consumers. The main consumer is Smith. It is an agent that analyzes domains looking for patterns. An investigator. Smith takes a set of analyzes to run, runs them and for each analysis generates a report that returns to the backend. Here the report is saved and possibly sent to the component which notifies appropriately.

To briefly understand these are the main components of the Matrix. The full list will come in a later post. However, what we want to share is the architectural approach. All these components communicate and to allow them to do so we could adopt dozens of solutions. At first we implemented everything as microservices basing communication on REST calls. The approach immediately proved to be inefficient. We then switched to using queues by adopting a broker. Operation was better and the amount of resources consumed by the broker was significant. We therefore decided to create a custom component. Thus Tramontana came to light. It is a server that moves and transforms data. Some concepts were taken from BizTalk, but this had to work and cost little 🙂

We then converted all the components to allow them to be consumed by Tramontana. The transformation was simple given that Tramontana monitors storage and moves files, possibly transforming them based on their destination. Each component then had to receive input in a file and produce results in files. These files would have been moved and transformed by Tramontana. This choice brings several benefits:

• The components were emptied of business logics linked to other components and transport.
• Traceability was centralized in a component, here in fact we can follow the flow of data from a single console.
• The components do their specific work without caring who will use their data.
• Tramontana reaches the components and therefore there is no need for this component to be published or exhibited.
• Everything is asynchronous. If a component stops, including the Tramontana itself, everything continues to work, even if partially since some data does not arrive; however, nothing stops and when the component is restarted everything returns to 100% operation.