CrowdStrike – Global Incident

Last Friday, July 19, the popular U.S. software company CrowdStrike caused a worldwide crash of Windows-based computers, impacting critical systems in banks, hospitals, transportation..resulting in a temporary disruption of daily operations.

The cause? An incorrect update of the Falcon Sensor AV/EDR platform, released as a configuration update at 04:09 UTC, which resulted in triggering a logical error on the OS resulting in a BSOD:

The content is a channel file located in the %WINDIR%\System32\drivers\CrowdStrike directory.

Channel file “C-00000291*.sys” with timestamp of 2024-07-19 0527 UTC or later is the reverted (good) version.

Channel file “C-00000291*.sys” with timestamp of 2024-07-19 0409 UTC is the problematic version.


The company subsequently released a procedure for identifying affected Windows clients, as well as a remediation plan, advising to perform a reboot and acquire the correct version of the file, on alternatively follow Microsoft’s procedure to enter safe mode and delete the affected file, or alternatively restore the system (beware of any Bitlocker Key present).

EDIT: an official recovery tool has also been released by Microsoft to automate the remediation process.

The problem has impacted not only physical PCs, but also Windows instances in the cloud. Below are some official links for remediation, such as Azure and AWS.

In any case, if the problem instances are protected with backup software, it is always possible to restore to the latest valid version.

This event reminds us of how today’s world is extremely technology-driven, and of the possible human errors that, if not limited, can lead to disastrous consequences.

VeeamON 2024 Recap

INTRO

Last week in Fort Lauderdale, Florida, VeeamON 2024 took place, as every year the most awaited and important event organized by Veeam Software.

This year’s event was particularly rich in announcements, and there was really no lack of surprises.

Lots of demos and technical sessions, though not all available for those like me who followed everything remotely.

Veeam’s vision continues to focus on data resilience through 5 main strategies: Data Backup, Data Freedom, Data Recovery, Data Security, and Data Intelligence.

WHAT’S NEW

Starting to explore Data Backup, the core part dedicated to protecting and saving data, new versions of some solutions were officially presented.

  • Oracle Linux Virtualization Manager (oVirt): native support for OLVM, a KVM-based virtualization platform, has been available already since a few weeks
  • Proxmox VE: announced a few weeks ago the compatibility with this virtualizer, during VeeamON 2024 the first demo was presented, with the Veeam solution that promises to be 3 times faster than the native backup solution. The official release is scheduled for next Q3 2024
  • VBA v7: some new features for the future version of Veeam Backup for Azure announced, including the introduction of support for Cosmos DB
  • Veeam Backup for AWS v8: new features also for the Amazon cloud backup solution, introducing, for example, support for Redshift and Fsx
  • VBM365 v8: many new features also for Veeam Backup for Microsoft 365, coming out probably next Q3 2024, including MFA for console, proxy pools, immutability for backup, restore operator audit in Veeam ONE
  • Veeam Backup for Salesforce v3: additional features for this solution as well, where support for data encryption, data archive and data pipeline will be introduced
  • K10 v7: of course, it could not be missed an overview of the new version of Kasten, which includes, among others, support for FIPS-Enabled Clusters, for Azure Blob Immutability and for VMs on Openshift

We now turn to the surprises, which, as anticipated, were not lacking. Notable among the new features announced were, without a doubt:

  • VBR server on Linux OS starting with v13, with the specific capabilities of native zero trust architecture, and support for HA of Config DB, which will add that level of resilience and automation to the software that to today was lacking
  • Enter ID Backup, a solution that will be intergrated into Veeam B&R, to protect data, such as users, groups and app registrations, from Microsoft’s cloud-based identity/access management solution (Q4 2024)
  • Mongo DB Plugin, which will increase the package of natively supported enterprise applications (Q3 2024)
  • Lenovo TruSacle Backup, which will intregrate Veeam Backup & Replication and Veeam ONE into Lenovo ThinkSystem solutions for on-premise backups


In addition, as we know, Veeam has recently expanded its range of solutions by introducing fully SaaS services, further explored at this three-day event, including:

  • Veeam Data Cloud for M365, a preconfigured Microsoft 365 backup solution with a predictable cost model (per user/unlimited space)
  • Veeam Data Cloud for Azure, native and optimized backup solution for Microsoft Azure
  • Veeam Vault, fully managed cloud storage, with flat/TB rates, including api call charges and any outbound traffic

We move on to the Data Freedom and Data Recovery strategies, which is Veeam’s ability to use its own format to move a piece of data from one platform to another, allowing it to bypass the so-called “vendor lock-in”.

In this section we can mention the announcement of more new features for the upcoming version of VRO (Veeam Recovery Orchestrator).

Regarding Data Security, that strategic component through which Veeam and its solutions help data to be resilient to increasingly frequent cyber attacks, much space was given to Coveware, a company specializing in incident response acquired by Veeam last April 2024.

In particular, the key role it can play in a Cyber Recovery phase was explored, as it offers services such as:

  • Assessment
  • Forensic Analysis
  • Identification of ransomware type and impact on the customer’s organization
  • Negotiation with cybercriminals
  • Incident remediation and documentation


Also in the area of Data Security, also worth mentioning is the new partnership with Palo Alto for SIEM integration.

Speaking of Data Intelligence, another big surprise presented was the formalization of the partnership with Microsoft for Copilot AI integration with Veeam solutions.

Finally, we must mention other improvements and developments announced on Veeam ONE, Veeam AI assistant, Linux Hardened Repository and Veeam Service Provider Console.

CONCLUSION

In short, there was a lot of news, and I’m sure there will be an opportunity to explore some of them in more detail in future posts..STAY TUNED! 💚

Veeam – Proxmox Announcement

This week Veeam Software made the much expected announcement: support for Proxmox will be released soon.

What is Proxmox, and why so much interest behind this news?

Proxmox VE (Virtual Environment) is an open-source, KVM-based virtualizer that allows both virtual machines and container-based architectures to run.

The recent acquisition of VMware by Broadcom, and the subsequent unknowns about the future strategies of the world leader in virtualization systems, have pushed many customers to look for possible alternatives to focus on for their infrastructures.

For this very reason, the name of Proxmox has gained popularity in the recent period, so much so that even Veeam has decided to focus on developing integration with this new hypervisor.

The first official demo will be presented at VeeamON 2024 to be held in Florida next June 3-5.

If you have not yet registered you can do it here.

Enjoy! 💚

Linux xz library vulnerability

Last Friday, a major vulnerability was reported on the xz library, used by some Linux distributions as a data compression program.

Specifically, the source code on Github was infected with malicious code properly obfuscated, allowing attackers to create a backdoor for ssh access to infected systems.

The CVE is currently listed by NIST with criticality 10.0, which is highest:

https://nvd.nist.gov/vuln/detail/CVE-2024-3094

The vulnerability, discovered almost accidentally by a Microsoft developer, is present in versions 5.6.0 – 5.6.1

Therefore, it is recommended to downgrade the xz library version on systems with this release, or to uninstall it if not in use.

Below is also the official note from Red Hat:

https://access.redhat.com/security/cve/CVE-2024-3094

WBD 2024

Today, March 31, is World Backup Day 2024!

This anniversary, which started way back in 2011, wants to make businesses as well as individuals aware of the importance of having their data backed up and safe with an annual commemoration.

If we think about any business in the world, small or large, we know for sure that every day it has to manage data to carry out its work. Personal registries, orders, payments are a few examples of activities that are indispensable for a business, activities that need to write this data in almost always digital devices, whether they are servers, storage or simple computers.

Let us now think of ourselves, our family members, our friends. Who among us does not use a smartphone or computer? Photos, videos, messages, important documents, all valuable material that we certainly don’t want to lose.

But what if the computer suddenly breaks down or our smartphone gets stolen? We would no longer be able to recover our data, unless we had backed it up first!

Well yes, for any important data, it is good practice to make at least a second copy and keep it in a safe place. Cloud, external hard drives, usb sticks are some examples of devices that can help us save our important files, a backup indeed!

Sometimes we do not realize the importance of something until we have lost it, but why take the risk?

Backup is the solution!

Microsoft OOB Updates

Microsoft released an important Out-Of-Band (OOB) update yesterday, which is an emergency fix to be installed before the upcoming April updates, for Windows Server versions 2022, 2016, and 2012 (not yet available for the 2019 version).

This update fixes a know issue that was identified in the last update in March: the problem afflicts the Active Directory Domain Controllers’ LSASS service, where a memory leak during kerberos authentication requests can cause the service to crash and unexpectedly reboot the server.

Microsoft recommends installing the update immediately in case your system falls into the described case scenario.

https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2022#3271msgdesc

Below are the reference KBs for the specific versions:

Windows Server 2022: KB5037422
Windows Server 2019: KB5037425
Windows Server 2016: KB5037423
Windows Server 2012 R2: KB5037426

EDIT: The update for Windows Server 2019 was realeased yesterday.