As you all know, during the last VeeamON 2024, the long-awaited Linux version for Veeam Backup Server installation was announced.
Those like me who are lucky enough to be part of the Veeam100 program, these days are having a chance to test the upcoming Veeam Data Platform v13 version.
Obviously this is a Technical Preview, so the future official version, which will be GA in Q2 2025, might vary a little in terms of user experience and implemented features.
Well, let’s start by revealing some details!
First, the TP looks like an OVA that can be installed on our hypervisor (e.g. VMWare).
According to current information, there should be three options for (Rocky) Linux-based software installation in the future:
Virtual Appliance (OVA/OVF)
Bootable ISO
Linux installable packages (rpm)
The first two options are obviously the most recommended, because they also include the operating system and are optimized as well as compliant with the DISA-STIG and FIPS standards.
Once the OVA is installed, access will be allowed only with non-root users.
Other information, following the principles of zero trust, even the software services are associated with non-privileged accounts. Regarding the console, the big news is the new colorful web interface, integrated into the Linux installation, which will initially operate alongside the classic VBR console.
This console will most likely not have immediately all the functionality of the classic VBR console installed with Windows, but it is still a good starting point to go and completely replace the “legacy sister”.
One option in my opinion very interesting introduced in this new console, is the possibility to manage software updates in a dedicated section, the Veeam Updater.
Very similar to what is already happening in dedicated Public Cloud backup appliances (Veeam for Azure/AWS/GCP), this section allows for custom updating of components, as well as a setting to force automatic deployment of security updates within a certain number of days of their availability.
Let us now turn to the preview of the VBR Windows console.
We can see lighter and more attractive graphics, with the option to enable dark mode as well.
In terms of new features, notable is the introduction of SAML authentication for integration with external providers and the Veeam Security Officer role (now Veeam Security Administrator) for managing the most sensitive operations.
We will stop here for now, waiting for the beta and of course the GA version! 💚
Let’s start from the beginning: security and backup.
Today, unfortunately, ransowmare attacks are on the rise, and defending against them is an increasingly difficult challenge.
If backups used to be considered as something not really important, perhaps useful only in case of any storage damage, today they have become the last resort to keep our data safe.
For this reason, one of the main targets during a cyber attack is the backup infrastructure: if threath actors succeed in taking it down, the road to ransom payment will be straight downhill.
News of collaborations and product integrations between large data protection and security vendors are now a daily occurrence, most recently the one between Veeam and Palo Alto Network Cortex XSIAM/XSOAR.
All this brings home to us how important it is to focus on the security of all systems, including backup infrastructure.
One of the several best practices recommended by Veeam, for example, is to try to make its components as anonymous as possible.
Assigning backup servers and repositories a name that cannot be identified with their role can be a first attempt to avoid making just about everything so easy for any malicious attackers.
Another method for attempting to identify and perhaps slow down an ongoing attack is to use honeypots: traps, decoys used to attract threat actors and draw them out.
The honeypot is a component that simulates the production system, possibly with the same applications, but with data that is not real.
In the case of Veeam Data Platform, the idea might be to create a VBR server that acts as a honeypot, perhaps even equipped with working backups.
Of course, this might require a not inconsiderable effort, because we would have to use sacrifiable, non-production systems, with the only purpose of attracting malicious attackers and having our anomaly detection software detect instrusion or tampering attempts on the honeypot.
This system simulates multiple Veeam and Windows services, such as Veeam Backup Server services, Veeam Hardened Repository, Veeam Windows Repository, Veeam Backup Enterprise Manager, SSH, RDP, Netbios.
It supports the use of multiple network cards, so each service can be associated with a specific VLAN, so it is ready for a realistic attack scenarios using lateral movement tactics (TA0008).
The system doesn’t receive any incoming traffic, so any connection attributable to the use of discovery tactics (TA0007) should represent an intrusion attempt.
This tool can be downloaded as an OVA appliance (compatible only with vSphere 8.0 or higher) or installed on a minimal Rocky Linux.
The console comes with a very simple yet comprehensive interface where we can manage the status of decoy services, associated network interfaces, and view real-time ports in use and connection logs on each specific service.
All captured connection attempts, including information such as source port, source ip, or credentials used, can be turned over to a centralized syslog or via email, so that alerting can be triggered and readily handled by a SOC.
Of course, we do not expect it to be our most effective weapon against cyber attacks, but in this battle between the two worlds it is still one more option! 💚
Last week in Fort Lauderdale, Florida, VeeamON 2024 took place, as every year the most awaited and important event organized by Veeam Software.
This year’s event was particularly rich in announcements, and there was really no lack of surprises.
Lots of demos and technical sessions, though not all available for those like me who followed everything remotely.
Veeam’s vision continues to focus on data resilience through 5 main strategies: Data Backup, Data Freedom, Data Recovery, Data Security, and Data Intelligence.
WHAT’S NEW
Starting to explore Data Backup, the core part dedicated to protecting and saving data, new versions of some solutions were officially presented.
Oracle Linux Virtualization Manager (oVirt): native support for OLVM, a KVM-based virtualization platform, has been available already since a few weeks
Proxmox VE: announced a few weeks ago the compatibility with this virtualizer, during VeeamON 2024 the first demo was presented, with the Veeam solution that promises to be 3 times faster than the native backup solution. The official release is scheduled for next Q3 2024
VBA v7: some new features for the future version of Veeam Backup for Azure announced, including the introduction of support for Cosmos DB
Veeam Backup for AWS v8: new features also for the Amazon cloud backup solution, introducing, for example, support for Redshift and Fsx
VBM365 v8: many new features also for Veeam Backup for Microsoft 365, coming out probably next Q3 2024, including MFA for console, proxy pools, immutability for backup, restore operator audit in Veeam ONE
Veeam Backup for Salesforce v3: additional features for this solution as well, where support for data encryption, data archive and data pipeline will be introduced
K10 v7: of course, it could not be missed an overview of the new version of Kasten, which includes, among others, support for FIPS-Enabled Clusters, for Azure Blob Immutability and for VMs on Openshift
We now turn to the surprises, which, as anticipated, were not lacking. Notable among the new features announced were, without a doubt:
VBR server on Linux OS starting with v13, with the specific capabilities of native zero trust architecture, and support for HA of Config DB, which will add that level of resilience and automation to the software that to today was lacking
Enter ID Backup, a solution that will be intergrated into Veeam B&R, to protect data, such as users, groups and app registrations, from Microsoft’s cloud-based identity/access management solution (Q4 2024)
Mongo DB Plugin, which will increase the package of natively supported enterprise applications (Q3 2024)
Lenovo TruSacle Backup, which will intregrate Veeam Backup & Replication and Veeam ONE into Lenovo ThinkSystem solutions for on-premise backups
In addition, as we know, Veeam has recently expanded its range of solutions by introducing fully SaaS services, further explored at this three-day event, including:
Veeam Data Cloud for M365, a preconfigured Microsoft 365 backup solution with a predictable cost model (per user/unlimited space)
Veeam Data Cloud for Azure, native and optimized backup solution for Microsoft Azure
Veeam Vault, fully managed cloud storage, with flat/TB rates, including api call charges and any outbound traffic
We move on to the Data Freedom and Data Recovery strategies, which is Veeam’s ability to use its own format to move a piece of data from one platform to another, allowing it to bypass the so-called “vendor lock-in”.
In this section we can mention the announcement of more new features for the upcoming version of VRO (Veeam Recovery Orchestrator).
Regarding Data Security, that strategic component through which Veeam and its solutions help data to be resilient to increasingly frequent cyber attacks, much space was given to Coveware, a company specializing in incident response acquired by Veeam last April 2024.
In particular, the key role it can play in a Cyber Recovery phase was explored, as it offers services such as:
Assessment
Forensic Analysis
Identification of ransomware type and impact on the customer’s organization
Negotiation with cybercriminals
Incident remediation and documentation
Also in the area of Data Security, also worth mentioning is the new partnership with Palo Alto for SIEM integration.
Speaking of Data Intelligence, another big surprise presented was the formalization of the partnership with Microsoft for Copilot AI integration with Veeam solutions.
Finally, we must mention other improvements and developments announced on Veeam ONE, Veeam AI assistant, Linux Hardened Repository and Veeam Service Provider Console.
CONCLUSION
In short, there was a lot of news, and I’m sure there will be an opportunity to explore some of them in more detail in future posts..STAY TUNED! 💚
This week Veeam Software made the much expected announcement: support for Proxmox will be released soon.
What is Proxmox, and why so much interest behind this news?
Proxmox VE (Virtual Environment) is an open-source, KVM-based virtualizer that allows both virtual machines and container-based architectures to run.
The recent acquisition of VMware by Broadcom, and the subsequent unknowns about the future strategies of the world leader in virtualization systems, have pushed many customers to look for possible alternatives to focus on for their infrastructures.
For this very reason, the name of Proxmox has gained popularity in the recent period, so much so that even Veeam has decided to focus on developing integration with this new hypervisor.
The first official demo will be presented at VeeamON 2024 to be held in Florida next June 3-5.
If you have not yet registered you can do it here.
When we talk about backup repositories in Veeam, we have to mention object storage, a technology that has been growing in popularity in recent years.
From version 12 of Veeam B&R, in fact, it is possible to directly write a backup to this type of repository.
Since version 12.1, it has also been possible to back up data stored on an object storage.
Unlike file system type storage architectures, which manage data hierarchically within directories, object storage architecture is flat, and is designed to store unstructured data, such as backups.
Specifically, the data is divided into blocks with associated metadata and unique identifiers, which are used by the system when accessing it.
The main advantages include that it can hold large amounts of data at no excessive cost, is easily scalable, and is compatible with HTTP/HTTPS and REST API protocols.
Wasabi is one of the cloud-based object storage vendors, so we can compare it to the better-known S3 from AWS or Azure Blob Storage from Microsoft.
Unlike the large vendors mentioned above, the price/TB is much lower, and there are no costs for ingress/egress traffic or API calls.
Wasabi is listed in the Veeam Ready compatibility directory as an object storage backup target (S3 compatible), and with native support for immutability (object lock) functionality.
The first thing to do to use Wasabi for our Veeam backups is to create a storage account by registering for the free 30-day trial; after that, it is possible to continue using the account in Pay As You Go or Reserved Capacity Storage mode.
Once registered and logged into the dashboard, generate a new access key/secret key pair, and create the bucket that will store our Veeam backups:
Now we can go to our Veeam B&R console, and from the main menu click on “Add Repository,” then select “Object Storage” and “Wasabi Cloud Storage”:
Once the wizard starts, enter the name we want to give on Veeam to our Wasabi repository:
Next, enter the details of the storage account and region on which we created our bucket:
At this point, enter the details of the bucket and folder to be used for our backups:
NB: for this tutorial in a lab environment the immutability flag was not enabled, but for production environments it is always recommended to use it
Finally, specify the mount server and complete the wizard:
Here is our Wasabi repository to use for our backup jobs:
Veeam ONE is Veeam software’s solution for monitoring virtual environments, such as vSphere, Vmware Cloud Director, Hyper-V, and data protection environments, such as Veeam Backup and Replication and Veeam Backup for Office 365.
As mentioned in a previous post, the latest VONE 12.1 release introduced the Veeam Threat Center dashboard: this tool allows us to view the overall security status of our VBRs, verifying compliance with the various best practices indicated by Veeam.
Specifically, the widgets we find are:
Data Platform Scorecard: shows an overall score of the health of our VBRs, defined by the parameters Platform Security Compliance, Data Recovery Health, Data Protection Status and Backup Immutability Status
Malware Detections: shows any malware or suspicious infections on our restore points
RPO Anomalies: shows objects that are out of range from the defined RPO
SLA Compliance Overview: highlights the percentage of achievement of our SLAs based on a period and success rate defined in the widget configuration
In order to take advantage of the potential of this dashboard, we must first add our VBR, making sure to also check the “Provide access to embedded dashboards” checkbox
Before configuration, within the VBR console the integration will not be active:
After configuration, the dashboard will be populated with the Veeam Threat Center view of Veeam ONE and other useful widgets.
Tip: when adding a VBR, pay attention to the compatibility of the licenses of the two products
As you may know, as of March 1, 2024, several outdated versions of Veeam products have gone into EOS (End of Support).
Examples of the most used products include Veeam B&R 11 and Veeam ONE 11:
To see the full list of Veeam product lifecycle visit the following link:
https://www.veeam.com/product-lifecycle.html
This week, a little surprisingly, a cumulative patch for Veeam Backup & Replication V11a was released:
https://www.veeam.com/kb4245
This update comes especially to those customers who due to usage requirements need to maintain compatibility with older hypervisors ( for example VMware vSphere/Esxi 5.5).
The patch contains some product fixes, and also some security fixes of third-party components included in the software, such as VDDK, OpenSSL, liblz4, zlib and Putty:
https://www.veeam.com/kb4245
Important note: If you decide to install this patch, you will no longer be able to upgrade to V12.1, but will have to wait for the release of the next minor update V12.2 (expected in the second half of 2024).
So if you are at V11 and have no compatibility issues with the rest of the infrastructure, the advice is to upgrade to the latest V12.1 version, taking advantage of the many added features immediately.
Below is one last link that may be useful when planning upgrades, the upgrade-path link for Veeam B&R.
Information, now mostly in the form of digital data, is a critical asset for all companies, from the smallest to the largest. The ISO/IEC 27001 standard reminds us what the requirements and best practices are for best managing the security of this information.
The three core principles are:
Confidentiality: not everyone can access a particular piece of private information, only people with the right permissions
Integrity of information: the data that the organization uses to conduct its business or that it keeps safe for others must be stored reliably, ensuring that it is not deleted or damaged
Availability of data: data must be available at all times, so that anyone with authorization can access the information whenever necessary
VEEAM’S ROLE
To protect this data, software solutions like Veeam Backup & Replication are crucial because they help to achieve the three mentioned cardinal principles of information security.
Specifically, Veeam allows us to:
create backups and replicas of our data, which means additional copies of the original information → help preserve integrity
keep backups protected from malicious action, hardware problems or disastrous natural events , leveraging immutability, air gapped and offsite copy→ helps keep the data always available
save our data through secure protocols and in an encrypted way→ helps maintain confidentiality
All this translates into Veeam’s fundamental rule, the famous 3-2-1-1-0. To this rule, indeed, I would add a property to be applied globally: encryption.
VEEAM ENCRYPTION – WHY AND HOW IT WORKS
Just like encryption on the original data, encryption of backups is not a practice that is always used, sometimes for reasons of “compatibility” with deduplication appliances, sometimes because we forget or do not consider it as necessary. In my opinion, however, it is one of the keys to ensuring the confidentiality of information. Whether we save backups on an external cloud or inside our datacenter, it is imperative to ensure that anyone with access to this data cannot read it unless authorized. Data exfiltration is something that can impact our backups as well, and if they are not encrypted any instance of VBR can read them.
Veeam provides both encryption in transit, that is, during the copy of the original data to the designated repository, and encryption at rest, that is, applied to the backup itself. Traffic encryption is based on TLS (since the latest version of Veeam v12.1, TLS 1.3 is also supported). Backup file encryption, on the other hand, is based on the Veeam Cryptographic Module and Microsoft Crypto API libraries, which are both FIPS compliant. To encrypt the data, a single-key encryption algorithm is used, which means a single key is used to encrypt and decrypt, leveraging the AES-256 standard.
Without going into too much detail about Cipher, KEX and so on, what I would like to describe is the hierarchical scheme and workflow of encryption in Veeam:
Starting from the bottom, we find:
session key: used on backup data blocks, changes with each backup session
metakey: used to encrypt backup metadata; like the session key, it changes with each backup session
storage key: the previous two keys are themselves encrypted by the storage key, which is used at the restore point level; in fact, when a backup chain is transformed and some backup data blocks are rewritten within a full ( for example, during syntetic full, reverse incremental, forever forward incremental.. operations), a single restore point will contain multiple session keys. The single storage key is able to act on the single restore point. It is maintained in the config db until the retention of the associated restore point expires.
user key: when the Veeam administrator creates an encryption password, and then enables encryption on a backup job, this password is used to generate the user key. This key, which acts at the job level itself, is used to encrypt the storage keys that will be generated for each individual restore point within the chain of this job
backup server keys: optional key pair, generated when connecting a backup server to the VBEM; according to the RSA asymmetric algorithm, the public key is passed to the VBEM, while the private key is kept in the VBR db. The key pair will be used to securely identify the backup server during any decryption request to the Enterprise Manager, according to the “password loss protection” feature
enterprise manager keys: optional key pair, generated when connecting a backup server to the VBEM; according to the RSA asymmetric algorithm, the public key is passed to the backup server, and it is used to encrypt the session keys in the same way as the user key; the private key is kept in the VBEM db and used in case of decryption, according to the “password loss protection” functionality
During a backup job so, along with the encrypted data blocks, the cryptograms of the session keys, metakey, storage key (one encrypted with the user key and one with the EM public key), user key, and EM public key are saved, which will then be used to identify the corresponding keys when performing a restore.
PASSWORD LOSS PROTECTION
As anticipated earlier, there is a feature in Veeam Enterprise Manager that allows a second chance to decrypt backups in case our backup server no longer has the password, for example, perhaps because they are old backups that had been removed from the configuration.
Prerequisites
VUL or socket licenses of at least Enterprise type
EM and original backup servers connected
As of Veeam 12.1, the password loss protection feature also supports integration with KMS. The key pair created by the EM is called a keyset. New keysets can be created, exported or imported. You can set the automatic generation of new keysets, and the retention period of them.
The passwordless restore process consists of the following steps:
1) the Veeam admin starts the “encryption key restore” process from the backup server 2) this wizard generates a request that contains, in an encrypted manner, the storage key and EM public key references used during backup to encrypt that data 3) the request is passed to the EM admin 4) EM admin starts the “password recovery” wizard in the EM and enters the received request 5) EM finds the corresponding keyset 6) EM, using the EM private key, decrypts the storage key and enters it into a response file 7) EM admin sends this response to the Veeam admin
8) the Veeam admin enters this response into the “encryption key restore” wizard, completing the decryption process
Limitations: if you lose the backup server, or the EM, or the EM keyset you will not be able to use the recovery procedure. The only way to be truly safe when using encryption is to never lose the user password. So, the basic rule is: SAVE THE ENCRYPTION PASSWORD SAFELY, perhaps applying the 3-2-1-1-0 golden rule even for this data!
CONCLUSION
In these times when cyber attacks are becoming more and more frequent, viewing backups as something secondary is a mistake not to be made; they should be viewed more as an indispensable extension of our data. Using best practices is strongly recommended..3-2-1-1-0 rule with encryption!
In a previous post, we went to explore the new and more interesting features of Veeam B&R version 12.1.
In this post we will go into more detail about the tool that allows us to keep an eye on the status of our backup infrastructure: the Security and Compliance Analyzer.
INTRODUCTION
When we design and implement our backup infrastructures, paying attention to security rules is now a must.
There are a number of general considerations that help us harden our servers, as well as many best practices that should be applied to our backups.
The new Security and Compliance Analyzer tool allows us to have just such a simple and intuitive overview of the implementation of these best practices on our backup server.
Let’s go through its functionality in detail.
THE TOOL
Access to the tool is clearly visible in the main bar of the Veeam Console:
As anticipated earlier, the checks are divided into two sections, “Backup Infrastructure Security” and “Product Configuration“.
BACKUP INFRASTRUCTURE SECURITY
As we can see, it is concerned with checking the implementation of certain best practices defined for the Windows operating system hosting our backup server.
The first settings that are recommended are to disable those services that are considered critical because they allow remote interaction with our server, which are “Remote Desktop,” “RemoteRegistry” and “Windows Remote Management“.
Then we move on to the rarely considered Windows Firewall: the best practice is to keep it active always, going to work with the inbound and outbound rules as needed. Note: Veeam B&R automatically creates the firewall rules necessary for its components to communicate with each other.
It is then recommended to disable the “WDigest credentials caching” and “Web Proxy Auto-Discovery service” features to prevent credential or MITM-type attacks.
The next check is on deprecated versions of SSL and TLS, such as SSL 2.0, SSL 3.0, TLS 1.0 and TLS 1.1, which should also be disabled.
As for potential script-based malware attacks, good practice is trying to limit them by disabling the “Windows Script Host“.
Returning to deprecated protocols, SMBv1 is also among those to be disabled, as it is affected by numerous vulnerabilities. Note: As of Windows Server 2016, it is disabled by default.
The last protocol to be disabled is the “Link-Local Multicast Name Resolution” to limit spoofing and MITM attacks.
Finally, security on the SMBv3 protocol is tested, checking that settings to prevent NTLMv2 relay type attacks are enabled.
PRODUCT CONFIGURATION
We now move on to controls on the software-side settings.
The strategies and configurations that Veeam recommends are obviously focused on preserving our backups.
MFA for the VBR console: since v12, multi-factor authentication can be enabled on the backup console.
Immutable or offline media: to protect backup files, we recommend using at least one repository with the data immutability feature enabled or media that can be disconnected from the network, such as tape or rotated drives.
Password loss protection: a setting in Veeam Enterprise Manager that allows us to decrypt our backup data in case the encryption password is lost.
Domain or non-domain?: Veeam recommends that we leave our server, and the other infrastructure components, at Workgroup. Note: In case we want to use join with AD, it is good practice to create a management domain dedicated exclusively to the backup environment.
Email notification: always remember to enable email notifications, it is essential to keep track of the outcome of backups and other events happening in the system.
3-2-1 rule: the golden rule advises us to have at least 3 copies of the data (including the original data, so two backup copies), on at least 2 different media and 1 offsite copy. Note: The rule has now evolved into 3-2-1-1-0, where the second 1 is the offline/immutable copy, and the 0 indicates the need to implement an automated validation procedure for our backups, and error-free during verification testing.
Reverse Incremental: is the method that produces more read and write operations on our repository, to be abandoned in favour of the standard incremental.
Unknown Linux servers: in case we need to add linux servers to our backup infrastructure, it is recommended to trust them manually rather than automatically.
Configuration Backup: as a best practice, the configuration backup should be saved to a repository external to the backup server itself. Note: from v12.1 you can select an immutable object storage repository for this task as well.
Proxy traffic encryption: if our virtual proxies use network transport mode, encryption (NBDSSL) is recommended.
Physical Hardened repository: to reduce the attack surface, the hardened type repository should reside on a physical server (and with local disks) instead of a virtual one.
Network traffic encryption: to enable secure communication in our backup network, both to the Internet and to private networks, it is recommended to globally enable encryption in the general software settings.
Linux authentication: best practice recommends that we do not use password-based authentication for our linux servers, but enter SSH through the use of the public-private key pair, preventing brute force and MITM type attacks.
Backup services: it is recommended to use “Local System” as the account for our Veeam services.
Configuration backup encryption: it is recommended to use encryption on Veeam’s configuration backup as well, for more secure management of sensitive data in the DB.
Password rotation: control is over the credentials of the various components added to our backup infrastructure and the encryption password, which should be changed at least once a year.
Hardened repository access: as a best practice, SSH on this type of repository should be disabled.
S3 object lock type: this check verifies that the immutability set on the S3s added on Veeam is Compliance type (not editable) and not Governance type (editable), going against any policies on data handling (e.g., GDPR) and of course the security of effective immutability of backups.
Backup encryption: especially if our backups are saved to a cloud repository, it is a good idea to enable encryption at the individual job level.
Latest updates: it is recommended to keep the Veeam B&R software updated to the latest release/patch.
IMPLEMENTATION
To facilitate the implementation of most of these best practices, Veeam has provided on its KBs a powershell script that fixes all 11 points related to backup infrastructure and 2 points related to product infrastructure, while the settings that need custom setups (such as, for example, setting up our mail server, choosing a repository, users for which to enable MFA, etc.) are obviously in the hands of the backup administrator to be configured manually.
The tool can be used both interactively and automatically.
It is possible, in fact, to set up a report with daily scheduling and emailing.
Finally, it is also possible to exclude one or more parameters from the controls by marking them as “suppressed“.
CONCLUSION
We will never tire of repeating how important security is, especially for backups, which are our last defense against loss or corruption of our data. This improved tool is a good starting point to help us keep things under control.
We conclude the post by reposting other useful links regarding security and best practices, with the hope that the Security and Compliance Analyzer will also be increasingly developed and improved according to the evolving guidelines.
