This anniversary, which started way back in 2011, wants to make businesses as well as individuals aware of the importance of having their data backed up and safe with an annual commemoration.
If we think about any business in the world, small or large, we know for sure that every day it has to manage data to carry out its work. Personal registries, orders, payments are a few examples of activities that are indispensable for a business, activities that need to write this data in almost always digital devices, whether they are servers, storage or simple computers.
Let us now think of ourselves, our family members, our friends. Who among us does not use a smartphone or computer? Photos, videos, messages, important documents, all valuable material that we certainly don’t want to lose.
But what if the computer suddenly breaks down or our smartphone gets stolen? We would no longer be able to recover our data, unless we had backed it up first!
Well yes, for any important data, it is good practice to make at least a second copy and keep it in a safe place. Cloud, external hard drives, usb sticks are some examples of devices that can help us save our important files, a backup indeed!
Sometimes we do not realize the importance of something until we have lost it, but why take the risk?
Microsoft released an important Out-Of-Band (OOB) update yesterday, which is an emergency fix to be installed before the upcoming April updates, for Windows Server versions 2022, 2016, and 2012 (not yet available for the 2019 version).
This update fixes a know issue that was identified in the last update in March: the problem afflicts the Active Directory Domain Controllers’ LSASS service, where a memory leak during kerberos authentication requests can cause the service to crash and unexpectedly reboot the server.
Microsoft recommends installing the update immediately in case your system falls into the described case scenario.
As you may know, as of March 1, 2024, several outdated versions of Veeam products have gone into EOS (End of Support).
Examples of the most used products include Veeam B&R 11 and Veeam ONE 11:
To see the full list of Veeam product lifecycle visit the following link:
https://www.veeam.com/product-lifecycle.html
This week, a little surprisingly, a cumulative patch for Veeam Backup & Replication V11a was released:
https://www.veeam.com/kb4245
This update comes especially to those customers who due to usage requirements need to maintain compatibility with older hypervisors ( for example VMware vSphere/Esxi 5.5).
The patch contains some product fixes, and also some security fixes of third-party components included in the software, such as VDDK, OpenSSL, liblz4, zlib and Putty:
https://www.veeam.com/kb4245
Important note: If you decide to install this patch, you will no longer be able to upgrade to V12.1, but will have to wait for the release of the next minor update V12.2 (expected in the second half of 2024).
So if you are at V11 and have no compatibility issues with the rest of the infrastructure, the advice is to upgrade to the latest V12.1 version, taking advantage of the many added features immediately.
Below is one last link that may be useful when planning upgrades, the upgrade-path link for Veeam B&R.
Information, now mostly in the form of digital data, is a critical asset for all companies, from the smallest to the largest. The ISO/IEC 27001 standard reminds us what the requirements and best practices are for best managing the security of this information.
The three core principles are:
Confidentiality: not everyone can access a particular piece of private information, only people with the right permissions
Integrity of information: the data that the organization uses to conduct its business or that it keeps safe for others must be stored reliably, ensuring that it is not deleted or damaged
Availability of data: data must be available at all times, so that anyone with authorization can access the information whenever necessary
VEEAM’S ROLE
To protect this data, software solutions like Veeam Backup & Replication are crucial because they help to achieve the three mentioned cardinal principles of information security.
Specifically, Veeam allows us to:
create backups and replicas of our data, which means additional copies of the original information → help preserve integrity
keep backups protected from malicious action, hardware problems or disastrous natural events , leveraging immutability, air gapped and offsite copy→ helps keep the data always available
save our data through secure protocols and in an encrypted way→ helps maintain confidentiality
All this translates into Veeam’s fundamental rule, the famous 3-2-1-1-0. To this rule, indeed, I would add a property to be applied globally: encryption.
VEEAM ENCRYPTION – WHY AND HOW IT WORKS
Just like encryption on the original data, encryption of backups is not a practice that is always used, sometimes for reasons of “compatibility” with deduplication appliances, sometimes because we forget or do not consider it as necessary. In my opinion, however, it is one of the keys to ensuring the confidentiality of information. Whether we save backups on an external cloud or inside our datacenter, it is imperative to ensure that anyone with access to this data cannot read it unless authorized. Data exfiltration is something that can impact our backups as well, and if they are not encrypted any instance of VBR can read them.
Veeam provides both encryption in transit, that is, during the copy of the original data to the designated repository, and encryption at rest, that is, applied to the backup itself. Traffic encryption is based on TLS (since the latest version of Veeam v12.1, TLS 1.3 is also supported). Backup file encryption, on the other hand, is based on the Veeam Cryptographic Module and Microsoft Crypto API libraries, which are both FIPS compliant. To encrypt the data, a single-key encryption algorithm is used, which means a single key is used to encrypt and decrypt, leveraging the AES-256 standard.
Without going into too much detail about Cipher, KEX and so on, what I would like to describe is the hierarchical scheme and workflow of encryption in Veeam:
Starting from the bottom, we find:
session key: used on backup data blocks, changes with each backup session
metakey: used to encrypt backup metadata; like the session key, it changes with each backup session
storage key: the previous two keys are themselves encrypted by the storage key, which is used at the restore point level; in fact, when a backup chain is transformed and some backup data blocks are rewritten within a full ( for example, during syntetic full, reverse incremental, forever forward incremental.. operations), a single restore point will contain multiple session keys. The single storage key is able to act on the single restore point. It is maintained in the config db until the retention of the associated restore point expires.
user key: when the Veeam administrator creates an encryption password, and then enables encryption on a backup job, this password is used to generate the user key. This key, which acts at the job level itself, is used to encrypt the storage keys that will be generated for each individual restore point within the chain of this job
backup server keys: optional key pair, generated when connecting a backup server to the VBEM; according to the RSA asymmetric algorithm, the public key is passed to the VBEM, while the private key is kept in the VBR db. The key pair will be used to securely identify the backup server during any decryption request to the Enterprise Manager, according to the “password loss protection” feature
enterprise manager keys: optional key pair, generated when connecting a backup server to the VBEM; according to the RSA asymmetric algorithm, the public key is passed to the backup server, and it is used to encrypt the session keys in the same way as the user key; the private key is kept in the VBEM db and used in case of decryption, according to the “password loss protection” functionality
During a backup job so, along with the encrypted data blocks, the cryptograms of the session keys, metakey, storage key (one encrypted with the user key and one with the EM public key), user key, and EM public key are saved, which will then be used to identify the corresponding keys when performing a restore.
PASSWORD LOSS PROTECTION
As anticipated earlier, there is a feature in Veeam Enterprise Manager that allows a second chance to decrypt backups in case our backup server no longer has the password, for example, perhaps because they are old backups that had been removed from the configuration.
Prerequisites
VUL or socket licenses of at least Enterprise type
EM and original backup servers connected
As of Veeam 12.1, the password loss protection feature also supports integration with KMS. The key pair created by the EM is called a keyset. New keysets can be created, exported or imported. You can set the automatic generation of new keysets, and the retention period of them.
The passwordless restore process consists of the following steps:
1) the Veeam admin starts the “encryption key restore” process from the backup server 2) this wizard generates a request that contains, in an encrypted manner, the storage key and EM public key references used during backup to encrypt that data 3) the request is passed to the EM admin 4) EM admin starts the “password recovery” wizard in the EM and enters the received request 5) EM finds the corresponding keyset 6) EM, using the EM private key, decrypts the storage key and enters it into a response file 7) EM admin sends this response to the Veeam admin
8) the Veeam admin enters this response into the “encryption key restore” wizard, completing the decryption process
Limitations: if you lose the backup server, or the EM, or the EM keyset you will not be able to use the recovery procedure. The only way to be truly safe when using encryption is to never lose the user password. So, the basic rule is: SAVE THE ENCRYPTION PASSWORD SAFELY, perhaps applying the 3-2-1-1-0 golden rule even for this data!
CONCLUSION
In these times when cyber attacks are becoming more and more frequent, viewing backups as something secondary is a mistake not to be made; they should be viewed more as an indispensable extension of our data. Using best practices is strongly recommended..3-2-1-1-0 rule with encryption!
The news we’ve been waiting for is finally here: with the release a few days ago of Esxi version 8.0 U2b, VMware confirms in the release notes that the bug that afflicted the Change Block Tracking (CBT) functionality in version 8.0 U2 has been fixed.
In vSphere 8.0 Update 2, to optimize the open and close process of virtual disks during hot extension, the disk remains open during hot extend operations. Due to this change, incremental backup of virtual disks with CBT enabled might be incomplete, because the CBT in-memory bitmap does not resize, and CBT cannot record the changes to the extended disk block. As a result, when you try to restore a VM from an incremental backup of virtual disks with CBT, the VM might fail to start.
The bug, documented in official KB 95965, afflicted virtual disks with CBT enabled and hot extended, and caused possible loss or corruption of data when restoring using backup software.
This is because, due to the bug, some information was not being properly updated in the CBT itself, which is the mechanism used just by backup software to read only blocks that have changed since the last incremental, without having to do a full disk scan each time.
Remember, however, that the patch does not fix any VMs that had run into the bug before this update was installed.
For this reason, it is necessary to apply the workaround recommended in the above KB, meaning to reset the CBT and run a new active full.
Finally, we recommend always performing restore tests (automated or manual) to make sure that our backups are valid.
In a previous post, we went to explore the new and more interesting features of Veeam B&R version 12.1.
In this post we will go into more detail about the tool that allows us to keep an eye on the status of our backup infrastructure: the Security and Compliance Analyzer.
INTRODUCTION
When we design and implement our backup infrastructures, paying attention to security rules is now a must.
There are a number of general considerations that help us harden our servers, as well as many best practices that should be applied to our backups.
The new Security and Compliance Analyzer tool allows us to have just such a simple and intuitive overview of the implementation of these best practices on our backup server.
Let’s go through its functionality in detail.
THE TOOL
Access to the tool is clearly visible in the main bar of the Veeam Console:
As anticipated earlier, the checks are divided into two sections, “Backup Infrastructure Security” and “Product Configuration“.
BACKUP INFRASTRUCTURE SECURITY
As we can see, it is concerned with checking the implementation of certain best practices defined for the Windows operating system hosting our backup server.
The first settings that are recommended are to disable those services that are considered critical because they allow remote interaction with our server, which are “Remote Desktop,” “RemoteRegistry” and “Windows Remote Management“.
Then we move on to the rarely considered Windows Firewall: the best practice is to keep it active always, going to work with the inbound and outbound rules as needed. Note: Veeam B&R automatically creates the firewall rules necessary for its components to communicate with each other.
It is then recommended to disable the “WDigest credentials caching” and “Web Proxy Auto-Discovery service” features to prevent credential or MITM-type attacks.
The next check is on deprecated versions of SSL and TLS, such as SSL 2.0, SSL 3.0, TLS 1.0 and TLS 1.1, which should also be disabled.
As for potential script-based malware attacks, good practice is trying to limit them by disabling the “Windows Script Host“.
Returning to deprecated protocols, SMBv1 is also among those to be disabled, as it is affected by numerous vulnerabilities. Note: As of Windows Server 2016, it is disabled by default.
The last protocol to be disabled is the “Link-Local Multicast Name Resolution” to limit spoofing and MITM attacks.
Finally, security on the SMBv3 protocol is tested, checking that settings to prevent NTLMv2 relay type attacks are enabled.
PRODUCT CONFIGURATION
We now move on to controls on the software-side settings.
The strategies and configurations that Veeam recommends are obviously focused on preserving our backups.
MFA for the VBR console: since v12, multi-factor authentication can be enabled on the backup console.
Immutable or offline media: to protect backup files, we recommend using at least one repository with the data immutability feature enabled or media that can be disconnected from the network, such as tape or rotated drives.
Password loss protection: a setting in Veeam Enterprise Manager that allows us to decrypt our backup data in case the encryption password is lost.
Domain or non-domain?: Veeam recommends that we leave our server, and the other infrastructure components, at Workgroup. Note: In case we want to use join with AD, it is good practice to create a management domain dedicated exclusively to the backup environment.
Email notification: always remember to enable email notifications, it is essential to keep track of the outcome of backups and other events happening in the system.
3-2-1 rule: the golden rule advises us to have at least 3 copies of the data (including the original data, so two backup copies), on at least 2 different media and 1 offsite copy. Note: The rule has now evolved into 3-2-1-1-0, where the second 1 is the offline/immutable copy, and the 0 indicates the need to implement an automated validation procedure for our backups, and error-free during verification testing.
Reverse Incremental: is the method that produces more read and write operations on our repository, to be abandoned in favour of the standard incremental.
Unknown Linux servers: in case we need to add linux servers to our backup infrastructure, it is recommended to trust them manually rather than automatically.
Configuration Backup: as a best practice, the configuration backup should be saved to a repository external to the backup server itself. Note: from v12.1 you can select an immutable object storage repository for this task as well.
Proxy traffic encryption: if our virtual proxies use network transport mode, encryption (NBDSSL) is recommended.
Physical Hardened repository: to reduce the attack surface, the hardened type repository should reside on a physical server (and with local disks) instead of a virtual one.
Network traffic encryption: to enable secure communication in our backup network, both to the Internet and to private networks, it is recommended to globally enable encryption in the general software settings.
Linux authentication: best practice recommends that we do not use password-based authentication for our linux servers, but enter SSH through the use of the public-private key pair, preventing brute force and MITM type attacks.
Backup services: it is recommended to use “Local System” as the account for our Veeam services.
Configuration backup encryption: it is recommended to use encryption on Veeam’s configuration backup as well, for more secure management of sensitive data in the DB.
Password rotation: control is over the credentials of the various components added to our backup infrastructure and the encryption password, which should be changed at least once a year.
Hardened repository access: as a best practice, SSH on this type of repository should be disabled.
S3 object lock type: this check verifies that the immutability set on the S3s added on Veeam is Compliance type (not editable) and not Governance type (editable), going against any policies on data handling (e.g., GDPR) and of course the security of effective immutability of backups.
Backup encryption: especially if our backups are saved to a cloud repository, it is a good idea to enable encryption at the individual job level.
Latest updates: it is recommended to keep the Veeam B&R software updated to the latest release/patch.
IMPLEMENTATION
To facilitate the implementation of most of these best practices, Veeam has provided on its KBs a powershell script that fixes all 11 points related to backup infrastructure and 2 points related to product infrastructure, while the settings that need custom setups (such as, for example, setting up our mail server, choosing a repository, users for which to enable MFA, etc.) are obviously in the hands of the backup administrator to be configured manually.
The tool can be used both interactively and automatically.
It is possible, in fact, to set up a report with daily scheduling and emailing.
Finally, it is also possible to exclude one or more parameters from the controls by marking them as “suppressed“.
CONCLUSION
We will never tire of repeating how important security is, especially for backups, which are our last defense against loss or corruption of our data. This improved tool is a good starting point to help us keep things under control.
We conclude the post by reposting other useful links regarding security and best practices, with the hope that the Security and Compliance Analyzer will also be increasingly developed and improved according to the evolving guidelines.
