Microsoft OOB Updates

Microsoft released an important Out-Of-Band (OOB) update yesterday, which is an emergency fix to be installed before the upcoming April updates, for Windows Server versions 2022, 2016, and 2012 (not yet available for the 2019 version).

This update fixes a know issue that was identified in the last update in March: the problem afflicts the Active Directory Domain Controllers’ LSASS service, where a memory leak during kerberos authentication requests can cause the service to crash and unexpectedly reboot the server.

Microsoft recommends installing the update immediately in case your system falls into the described case scenario.

https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2022#3271msgdesc

Below are the reference KBs for the specific versions:

Windows Server 2022: KB5037422
Windows Server 2019: KB5037425
Windows Server 2016: KB5037423
Windows Server 2012 R2: KB5037426

EDIT: The update for Windows Server 2019 was realeased yesterday.

Veeam 11a Patch – EOS

As you may know, as of March 1, 2024, several outdated versions of Veeam products have gone into EOS (End of Support).

Examples of the most used products include Veeam B&R 11 and Veeam ONE 11:

To see the full list of Veeam product lifecycle visit the following link:

https://www.veeam.com/product-lifecycle.html

This week, a little surprisingly, a cumulative patch for Veeam Backup & Replication V11a was released:

https://www.veeam.com/kb4245

This update comes especially to those customers who due to usage requirements need to maintain compatibility with older hypervisors ( for example VMware vSphere/Esxi 5.5).

The patch contains some product fixes, and also some security fixes of third-party components included in the software, such as VDDK, OpenSSL, liblz4, zlib and Putty:

https://www.veeam.com/kb4245

Important note: If you decide to install this patch, you will no longer be able to upgrade to V12.1, but will have to wait for the release of the next minor update V12.2 (expected in the second half of 2024).

So if you are at V11 and have no compatibility issues with the rest of the infrastructure, the advice is to upgrade to the latest V12.1 version, taking advantage of the many added features immediately.

Below is one last link that may be useful when planning upgrades, the upgrade-path link for Veeam B&R.

https://www.veeam.com/kb2053

Enjoy! πŸ’š

Veeam Encryption (What) is the key ?

INTRODUCTION

Information, now mostly in the form of digital data, is a critical asset for all companies, from the smallest to the largest.
The ISO/IEC 27001 standard reminds us what the requirements and best practices are for best managing the security of this information.

The three core principles are:

  • Confidentiality: not everyone can access a particular piece of private information, only people with the right permissions
  • Integrity of information: the data that the organization uses to conduct its business or that it keeps safe for others must be stored reliably, ensuring that it is not deleted or damaged
  • Availability of data: data must be available at all times, so that anyone with authorization can access the information whenever necessary

VEEAM’S ROLE

To protect this data, software solutions like Veeam Backup & Replication are crucial because they help to achieve the three mentioned cardinal principles of information security.

Specifically, Veeam allows us to:

  • create backups and replicas of our data, which means additional copies of the original information β†’ help preserve integrity
  • keep backups protected from malicious action, hardware problems or disastrous natural events , leveraging immutability, air gapped and offsite copyβ†’ helps keep the data always available
  • save our data through secure protocols and in an encrypted wayβ†’ helps maintain confidentiality


All this translates into Veeam’s fundamental rule, the famous 3-2-1-1-0.
To this rule, indeed, I would add a property to be applied globally: encryption.

VEEAM ENCRYPTION – WHY AND HOW IT WORKS

Just like encryption on the original data, encryption of backups is not a practice that is always used, sometimes for reasons of “compatibility” with deduplication appliances, sometimes because we forget or do not consider it as necessary.
In my opinion, however, it is one of the keys to ensuring the confidentiality of information.
Whether we save backups on an external cloud or inside our datacenter, it is imperative to ensure that anyone with access to this data cannot read it unless authorized.
Data exfiltration is something that can impact our backups as well, and if they are not encrypted any instance of VBR can read them.

Veeam provides both encryption in transit, that is, during the copy of the original data to the designated repository, and encryption at rest, that is, applied to the backup itself.
Traffic encryption is based on TLS (since the latest version of Veeam v12.1, TLS 1.3 is also supported).
Backup file encryption, on the other hand, is based on the Veeam Cryptographic Module and Microsoft Crypto API libraries, which are both FIPS compliant.
To encrypt the data, a single-key encryption algorithm is used, which means a single key is used to encrypt and decrypt, leveraging the AES-256 standard.

Without going into too much detail about Cipher, KEX and so on, what I would like to describe is the hierarchical scheme and workflow of encryption in Veeam:


Starting from the bottom, we find:

  • session key: used on backup data blocks, changes with each backup session
  • metakey: used to encrypt backup metadata; like the session key, it changes with each backup session
  • storage key: the previous two keys are themselves encrypted by the storage key, which is used at the restore point level; in fact, when a backup chain is transformed and some backup data blocks are rewritten within a full ( for example, during syntetic full, reverse incremental, forever forward incremental.. operations), a single restore point will contain multiple session keys. The single storage key is able to act on the single restore point. It is maintained in the config db until the retention of the associated restore point expires.
  • user key: when the Veeam administrator creates an encryption password, and then enables encryption on a backup job, this password is used to generate the user key. This key, which acts at the job level itself, is used to encrypt the storage keys that will be generated for each individual restore point within the chain of this job
  • backup server keys: optional key pair, generated when connecting a backup server to the VBEM; according to the RSA asymmetric algorithm, the public key is passed to the VBEM, while the private key is kept in the VBR db. The key pair will be used to securely identify the backup server during any decryption request to the Enterprise Manager, according to the “password loss protection” feature
  • enterprise manager keys: optional key pair, generated when connecting a backup server to the VBEM; according to the RSA asymmetric algorithm, the public key is passed to the backup server, and it is used to encrypt the session keys in the same way as the user key; the private key is kept in the VBEM db and used in case of decryption, according to the “password loss protection” functionality


During a backup job so, along with the encrypted data blocks, the cryptograms of the session keys, metakey, storage key (one encrypted with the user key and one with the EM public key), user key, and EM public key are saved, which will then be used to identify the corresponding keys when performing a restore.

PASSWORD LOSS PROTECTION

As anticipated earlier, there is a feature in Veeam Enterprise Manager that allows a second chance to decrypt backups in case our backup server no longer has the password, for example, perhaps because they are old backups that had been removed from the configuration.

Prerequisites

  • VUL or socket licenses of at least Enterprise type
  • EM and original backup servers connected

As of Veeam 12.1, the password loss protection feature also supports integration with KMS.
The key pair created by the EM is called a keyset. New keysets can be created, exported or imported.
You can set the automatic generation of new keysets, and the retention period of them.

The passwordless restore process consists of the following steps:

1) the Veeam admin starts the “encryption key restore” process from the backup server
2) this wizard generates a request that contains, in an encrypted manner, the storage key and EM public key references used during backup to encrypt that data
3) the request is passed to the EM admin
4) EM admin starts the “password recovery” wizard in the EM and enters the received request
5) EM finds the corresponding keyset
6) EM, using the EM private key, decrypts the storage key and enters it into a response file
7) EM admin sends this response to the Veeam admin

8) the Veeam admin enters this response into the “encryption key restore” wizard, completing the decryption process

Limitations: if you lose the backup server, or the EM, or the EM keyset you will not be able to use the recovery procedure.
The only way to be truly safe when using encryption is to never lose the user password.
So, the basic rule is: SAVE THE ENCRYPTION PASSWORD SAFELY, perhaps applying the 3-2-1-1-0 golden rule even for this data!

CONCLUSION

In these times when cyber attacks are becoming more and more frequent, viewing backups as something secondary is a mistake not to be made; they should be viewed more as an indispensable extension of our data.
Using best practices is strongly recommended..3-2-1-1-0 rule with encryption!

REFERENCES

https://helpcenter.veeam.com/docs/backup/vsphere/data_encryption.html?ver=120
https://helpcenter.veeam.com/docs/backup/em/em_manage_keys.html?ver=120 ​

Esxi 8.0 U2b – CBT Bug Fixed

The news we’ve been waiting for is finally here: with the release a few days ago of Esxi version 8.0 U2b, VMware confirms in the release notes that the bug that afflicted the Change Block Tracking (CBT) functionality in version 8.0 U2 has been fixed.

In vSphere 8.0 Update 2, to optimize the open and close process of virtual disks during hot extension, the disk remains open during hot extend operations. Due to this change, incremental backup of virtual disks with CBT enabled might be incomplete, because the CBT in-memory bitmap does not resize, and CBT cannot record the changes to the extended disk block. As a result, when you try to restore a VM from an incremental backup of virtual disks with CBT, the VM might fail to start.

This issue is resolved in this release.

https://docs.vmware.com/en/VMware-vSphere/8.0/rn/vsphere-esxi-80u2b-release-notes/index.html

The bug, documented in official KB 95965, afflicted virtual disks with CBT enabled and hot extended, and caused possible loss or corruption of data when restoring using backup software.

https://kb.vmware.com/s/article/95965

This is because, due to the bug, some information was not being properly updated in the CBT itself, which is the mechanism used just by backup software to read only blocks that have changed since the last incremental, without having to do a full disk scan each time.

Remember, however, that the patch does not fix any VMs that had run into the bug before this update was installed.

For this reason, it is necessary to apply the workaround recommended in the above KB, meaning to reset the CBT and run a new active full.

Finally, we recommend always performing restore tests (automated or manual) to make sure that our backups are valid.

Veeam v12.1 – Security and Compliance Analyzer

In a previous post, we went to explore the new and more interesting features of Veeam B&R version 12.1.

In this post we will go into more detail about the tool that allows us to keep an eye on the status of our backup infrastructure: the Security and Compliance Analyzer.

INTRODUCTION

When we design and implement our backup infrastructures, paying attention to security rules is now a must.

There are a number of general considerations that help us harden our servers, as well as many best practices that should be applied to our backups.

The new Security and Compliance Analyzer tool allows us to have just such a simple and intuitive overview of the implementation of these best practices on our backup server.

Let’s go through its functionality in detail.

THE TOOL

Access to the tool is clearly visible in the main bar of the Veeam Console:

As anticipated earlier, the checks are divided into two sections, “Backup Infrastructure Security” and “Product Configuration“.

BACKUP INFRASTRUCTURE SECURITY

As we can see, it is concerned with checking the implementation of certain best practices defined for the Windows operating system hosting our backup server.

  • The first settings that are recommended are to disable those services that are considered critical because they allow remote interaction with our server, which are “Remote Desktop,” “Remote Registry” and “Windows Remote Management“.

  • Then we move on to the rarely considered Windows Firewall: the best practice is to keep it active always, going to work with the inbound and outbound rules as needed. Note: Veeam B&R automatically creates the firewall rules necessary for its components to communicate with each other.

  • It is then recommended to disable the “WDigest credentials caching” and “Web Proxy Auto-Discovery service” features to prevent credential or MITM-type attacks.

  • The next check is on deprecated versions of SSL and TLS, such as SSL 2.0, SSL 3.0, TLS 1.0 and TLS 1.1, which should also be disabled.

  • As for potential script-based malware attacks, good practice is trying to limit them by disabling the “Windows Script Host“.

  • Returning to deprecated protocols, SMBv1 is also among those to be disabled, as it is affected by numerous vulnerabilities. Note: As of Windows Server 2016, it is disabled by default.
  • The last protocol to be disabled is the “Link-Local Multicast Name Resolution” to limit spoofing and MITM attacks.

  • Finally, security on the SMBv3 protocol is tested, checking that settings to prevent NTLMv2 relay type attacks are enabled.

PRODUCT CONFIGURATION

We now move on to controls on the software-side settings.

The strategies and configurations that Veeam recommends are obviously focused on preserving our backups.

  • MFA for the VBR console: since v12, multi-factor authentication can be enabled on the backup console.

  • Immutable or offline media: to protect backup files, we recommend using at least one repository with the data immutability feature enabled or media that can be disconnected from the network, such as tape or rotated drives.

  • Password loss protection: a setting in Veeam Enterprise Manager that allows us to decrypt our backup data in case the encryption password is lost.

  • Domain or non-domain?: Veeam recommends that we leave our server, and the other infrastructure components, at Workgroup. Note: In case we want to use join with AD, it is good practice to create a management domain dedicated exclusively to the backup environment.

  • Email notification: always remember to enable email notifications, it is essential to keep track of the outcome of backups and other events happening in the system.

  • 3-2-1 rule: the golden rule advises us to have at least 3 copies of the data (including the original data, so two backup copies), on at least 2 different media and 1 offsite copy. Note: The rule has now evolved into 3-2-1-1-0, where the second 1 is the offline/immutable copy, and the 0 indicates the need to implement an automated validation procedure for our backups, and error-free during verification testing.
  • Reverse Incremental: is the method that produces more read and write operations on our repository, to be abandoned in favour of the standard incremental.

  • Unknown Linux servers: in case we need to add linux servers to our backup infrastructure, it is recommended to trust them manually rather than automatically.

  • Configuration Backup: as a best practice, the configuration backup should be saved to a repository external to the backup server itself. Note: from v12.1 you can select an immutable object storage repository for this task as well.

  • Proxy traffic encryption: if our virtual proxies use network transport mode, encryption (NBDSSL) is recommended.

  • Physical Hardened repository: to reduce the attack surface, the hardened type repository should reside on a physical server (and with local disks) instead of a virtual one.

  • Network traffic encryption: to enable secure communication in our backup network, both to the Internet and to private networks, it is recommended to globally enable encryption in the general software settings.
  • Linux authentication: best practice recommends that we do not use password-based authentication for our linux servers, but enter SSH through the use of the public-private key pair, preventing brute force and MITM type attacks.

  • Backup services: it is recommended to use “Local System” as the account for our Veeam services.

  • Configuration backup encryption: it is recommended to use encryption on Veeam’s configuration backup as well, for more secure management of sensitive data in the DB.

  • Password rotation: control is over the credentials of the various components added to our backup infrastructure and the encryption password, which should be changed at least once a year.

  • Hardened repository access: as a best practice, SSH on this type of repository should be disabled.

  • S3 object lock type: this check verifies that the immutability set on the S3s added on Veeam is Compliance type (not editable) and not Governance type (editable), going against any policies on data handling (e.g., GDPR) and of course the security of effective immutability of backups.

  • Backup encryption: especially if our backups are saved to a cloud repository, it is a good idea to enable encryption at the individual job level.

  • Latest updates: it is recommended to keep the Veeam B&R software updated to the latest release/patch.

IMPLEMENTATION

To facilitate the implementation of most of these best practices, Veeam has provided on its KBs a powershell script that fixes all 11 points related to backup infrastructure and 2 points related to product infrastructure, while the settings that need custom setups (such as, for example, setting up our mail server, choosing a repository, users for which to enable MFA, etc.) are obviously in the hands of the backup administrator to be configured manually.

Below is the link to download the script: https://www.veeam.com/kb4525

USE

The tool can be used both interactively and automatically.

It is possible, in fact, to set up a report with daily scheduling and emailing.

Finally, it is also possible to exclude one or more parameters from the controls by marking them as “suppressed“.

CONCLUSION

We will never tire of repeating how important security is, especially for backups, which are our last defense against loss or corruption of our data. This improved tool is a good starting point to help us keep things under control.

We conclude the post by reposting other useful links regarding security and best practices, with the hope that the Security and Compliance Analyzer will also be increasingly developed and improved according to the evolving guidelines.

https://helpcenter.veeam.com/docs/backup/vsphere/security_guidelines.html?ver=120

https://bp.veeam.com/security

https://go.veeam.com/rs/870-LBG-312/images/veeam-security-checklist.pdf

https://go.veeam.com/rs/870-LBG-312/images/veeam-security-best-practices-2022.pdf

https://community.veeam.com/cyber-security-space-95/hardening-veeam-12-server-the-definitive-checklist-4255

Enjoy! πŸ’š

Veeam v12.1 – What’s New

In this post we are going to describe in general way the new and main features of the latest Veeam Data Platform 12.1 release.


Without a doubt, the main skill added to the software engine is Malware Detection, that is the ability to detect and identify cyber attacks, by leveraging three new technologies:


Inline malware detection: based on ML (Machine Learning) methods, it performs real-time, low-impact analysis of the backup stream to detect possible encryption activities taking place on the data


Suspicious file system activity detection: searches, by indexing the guest file system, for suspicious files, such as known malware extensions, ransom notes, etc.; it also analyzes file system activity, comparing previous indexes in order to detect suspicious changes, such as on the number and type of files present

Early threat detection: takes advantage of the Veeam Incident API to receive notifications from EDR/XDR about possible infections taking place on servers in our infrastructure; this allows Veeam B&R to mark corresponding subsequent backups as compromised; it is also possible to trigger an automatic backup to the infected server as a response to this event, so that we try to secure as many files as possible before the encryption task is completed

The second important aspect concerns the ability to respond to a possible malware attack more quickly and efficiently. The features that can perform this important innovation are:


Scan backups with YARA: in addition to the classic scan with antivirus, in this version Veeam has introduced the possibility, in order to perform checks during the restore phase, to ultilize also the YARA rules, parts of code based on specific patterns depending on the type of search or the files to be found (for example, for a particular family of malware); the scan is now able to search more quickly, in a sequential or binary manner, for an non-infected backup file, speeding up the restore operations following an attack; it is also possible to use SureBackup jobs in scan-only mode (without Virtual Lab)

Avoid reinfection with threat tracking: in this new version, the software is able to detect and keep track of which backups are potentially infected, so as to avoid any restore of already compromised files; in case of false positives, an exclusion can be set manually


Event forwarding: with the introduction of Syslog support, Veeam is able to send any event to a SIEM of our choice, so as to trigger mechanisms to react to certain security incidents reported by the software

Finally, the security and compliance of certain operations has been improved.


Four-eyes authorization: a setting that activates a double-check on particular sensitive operations, such as deleting a backup, a repository or adding a new Veeam Administrator, allowing to limit accidental errors or compromise attempts by a malicious user; specifically, when an admin performs one of these operations, a second admin’s approval is required within a configurable time range, after which the request is rejected

Key Management Server (KMS) integration: thanks to the integration with KMIP (Key Management Interoperability Protocol), it is now possible to use any supported KMS to perform automatic rotation of encryption keys


Security and compliance analyzer: a tool built into the VBR console, it allows for manual or scheduled verification of compliance with specific security baselines of our backup infrastructure, ensuring that various software best practices are being applied; it has been improved over v12, introducing many more controls, and enabling the ability to schedule a report and send it via email

Veeam Threat Center: a specific Veeam ONE dashboard is now integrated into the VBR console, and allows us to highlight identified malicious events, possible risks and critical areas, as well as a score on the overall status of our backup infrastructure based on the implementation of various best practices recommended by the software

Other important features added are:


Object storage backup: thanks to a storage-agnostic architecture, the ability to backup object storage type sources has been included, protecting the data in our buckets, whether they are on-prem or in the cloud


CDP engine enhancement: the Veeam Continous Data Protection, which allows for the smallest RPOs for our backups, has been improved both in terms of functionality (4x number of VM-vDisks supported) and efficiency (reduced computational requirements by 2x); also introduced the ability to perform failover tests without interrupting current replicas


Veeam AI assistant: here within the VBR console is our “personal assistant” based on the OpenAI model, which can be used, thanks to its learning from official Veeam documentation, for help and advice on our backup infrastructure

As soon as possible, future posts will explore some of these new features individually.


For details of all the many features introduced with Veeam Data Platform 12.1 please refer to the following official document.

https://www.veeam.com/veeam_backup_12_1_whats_new_wn.pdf


Enjoy! πŸ’š

How To – Upgrade to Veeam Data Platform 12.1

Since last December 2023, the ISO with the latest software version 12.1 is available for download on the Veeam website.

https://www.veeam.com/download-version.html

In order to update software in a safe and controlled manner, a few essential aspects should be evaluated first.

PREPARATION ACTIVITIES

LICENSE

In order to proceed with the upgrade, it is necessary to first perform a license validity check, which means that the support contract must be active and not expired.

REQUIREMENTS AND COMPATIBILITY MATRIX

The second step is to check the minimum compatibility requirements of the various systems/components that interact with our backup environment, such as: the backup server, proxies, hypervisors, backup repositories, etc..

For all these components, we need to make sure that all their hardware and software specifications are in matrix with the new version of Veeam 12.1 ( for example, vCenter and Esxi versions must be at least at 6.x)

https://helpcenter.veeam.com/docs/backup/vsphere/system_requirements.html?ver=120

https://helpcenter.veeam.com/docs/backup/vsphere/platform_support.html?ver=120

UPGRADE PATH

It is also necessary to check the upgrade path of the software itself, in other words, what is the minimum version we need to have in order to upgrade it directly without having to take multiple version steps: in this case, in order to upgrade to the latest build of12.1 (build 12.1.1.56), we need to have at least between version 10a (10.0.1.4854) and version 12 (12.0.0.1420 P20230718) .

PORTS AND PERMISSIONS

As an additional tip, it is always a good idea to double-check the ports on which the various components will need to communicate and the permissions configured to perform all operations, since, especially for major version jumps, there may have been changes over time on the requirements needed.

https://helpcenter.veeam.com/docs/backup/vsphere/used_ports.html?ver=120

https://helpcenter.veeam.com/docs/backup/vsphere/required_permissions.html?ver=120

For more detailed information, you can find the complete checklist with all of Veeam’s recommended checks in the official helpcenter or in the release notes document.

https://helpcenter.veeam.com/docs/backup/vsphere/upgrade_vbr_byb.html?ver=120

https://www.veeam.com/veeam_backup_12_1_release_notes_rn.pdf

PRE-UPGRADE

As operational steps before starting the upgrade, remember to:

  • check if the last backup session ended successfully
  • disable all jobs; if there is anything running, possibly wait for completion
  • backup SQL database
  • backup Veeam configuration
  • take a snapshot of the backup server (if it is virtual)

Ok, let’s start now with the installation!

UPGRADE WIZARD

  1. Start setup

  1. Select the product to upgrade

  1. Read and accept license agreement

  1. Verify all components to upgrade

  1. Select a valid license file

  1. Install, if necessary, any missing component

  1. Specify service account

  1. Specify SQL instance and DB name

  1. Confirm whether the database will be upgraded

  1. Start the upgrade

  1. Finish

POST-UPGRADE

After the upgrade is complete, open the console, follow the remote component upgrade wizard (if not selected during upgrade) and reactivate previously disabled jobs.

Official upgrade reference with more details here:

https://helpcenter.veeam.com/docs/backup/vsphere/upgrade_vbr.html?ver=120

Enjoy! πŸ’š