Infosecurity.US

News, overnight, of the apparent destruction, and subsequent data loss, of an estimated one hundred thousand websites hosted in the United Kingdom at VASERV.

Evidence point to a virtulization exploit – at least from the perspective of the ISP (The truth probably resides somewhere between appalling poor security assessment/management/policies and vulnerabilities in their hosted, virtualized platforms,); notwithstanding, we do sympathize with the administrators tasked with bringing these system back online, not to mention the unfortunate data owners. Backups, we don’t need no stinkin’ backups! Woops…

More information (comprised of the original post at The Register, as well as a status page from VASERV, tell the sorry tale) and links appears after the jump.

via El Reg’s Dan Goodin: “Webhost hack wipes out data for 100,000 sites“

“Vaserv suspects zero-day virtualization vuln”

“A large internet service provider said data for as many as 100,000 websites was destroyed by attackers who targeted a zero-day vulnerability in a widely-used virtualization application…” “Technicians at UK-based Vaserv.com were still scrambling to recover data on Monday evening UK time, more than 24 hours after unknown hackers were able to gain root access to the company’s system, Rus Foster, the company’s director told The Register. He said the attackers were able to penetrate his servers by exploiting a critical vulnerability in HyperVM, a virtualization application made by a company called LXLabs…”

—

Old status on index 1 | index 2

FSCKVPS Summary

The following fsck servers have total data loss and we will be inspecting them further later
8 9 13 15 17 21 22 23 25 26 27 30 34 36 41 43 44 45 46 48 50 51 52 54

All other servers should be online.

VAServ Status

US

VZ US West Coast – some of the nodes have been restored while others are being checked and once done full list will be issued
VZ TX – We are still working on either investigating these or having them in the queue for investigation

UK

The following nodes have been restored:
vz1-vz9uk / vz40uk / vz49uk
Beyond that the following servers are still being worked on
vz26uk / vz36uk / vz37uk / vz42uk / vz61uk / vz62uk / vz63uk / vz65uk / vz67uk / vz72uk / vz73uk / vz74uk / vz75uk

If your server is not on the list of our status page that either means:

• it is being checked currently
• it is pending for a proper investigation and is in the queue

In case you need support please raise a ticket with http://support.vaserv.com/ but kindly consider high volume of tickets and kindly open tickets for top priority inquiries for the moment. Your patience is well appreciated.

News:
22:19 vz47uk restored
22:21 vz46uk data loss
22:42 Please allow upto 2 hours for a ticket response as currently we have 200+ active tickets
23:02 vz67uk data loss
23:20 vz50uk data restored
23:23 vz51uk data loss
23:43 We currently have people onsite restoring UK servers and installing new servers for the people who have elected to have clean installs. We still have staff working through open support tickets. We will be continuting server restores in approx 90 minutes
00:03 FsckVPS server26 and server27 are still being worked on, but data *appears* to be intact
00:18 FsckVPS server33 is being worked on
00:45 FsckVPS server33 has approximately 50% of data lost. 7 VPSes on this box remain intact.
01:11 Fsckvps server48 has suffered from a complete data loss
01:15 Fsckvps ns1.fsckvps.com and ns2.fsckvps.com are currently being worked on and will be restored if possible.
03:46 Fsckvps at this time server54 and server52 are still under reconstruction/investigation and no further information is available regarding them
04:14 vz20uk no data
04:16 vz31uk fixed
04:26 vz74uk no data
04:31 vz75uk no data
04;44 vz43uk restored
04:56 vz60uk no data
05:08 FsckVPS server26, server27, server52, server54 all 100% data lost.
05:09 FsckVPS server33 being worked on, DNS being worked on.
05:21 FsckVPS dns servers will be alive in an eta of 15 minutes. (basic dns service).
05:32 FSCKVPS DNS is now resyncing
05:33 A general status report now follows FSCKVPS – Its about as good as its going to get so we are now starting to tackle individual issues
VAServ West Coast – Restored as its going to get
Texas – We are waiting for TMS to finish reloading. We have 7 or so servers still down there VAServ Atlanta – We still have vzspecial* to go over but other servers in the DC are up
VAServ UK – We have about 10 servers left to do
To place this into context in the last 36 hours we have recommissioned approx 180 servers and restored approx 2000 VPS using a standard config. If you are seeing disk space/memory issues please let us know via ticket. We will be going round sorting out correct quotas when things are quieter but for now people will be given allocations that would generally be above their default just so they can work.i
05:48: New/replacement builds. We have approx 300 customers wanting new VPS. We are going to start building these on the UK servers BSQ have lent us
07:42 vz26uk restored
07:55 vz14tx restored
07:56 vz2tx data loss
08:08 vz4tx restored
08:16 vz6x restored
08:19 vz8tx data loss
13:00 vzspecial1,2,4,5,7 restored 3,6,8 has suffered a data loss
13:01 vz37uk restored
13:20 vz63uk.vaserv.com restored
13:30 vz72 restored
We are going to finish up the final few UK nodes then go over all the other nodes top to bottom to clean out any problems that have arisen. Please excuse the spelling as currently most of us have been pulling double/triple shifts.
13:33 vz65uk restored
15:40 vz10uk has been reloaded and complete data loss has been determined
17:32 Dear Customer,

We have worked tirelessly through the night and over the last 48 hours to recover as many VPS as possible. However, we have now reached the end of all of our servers, and as such, if your server is not currently up, or not partly up (i.e. it is up but not working due to a configuration issue) then it is unfortunate that you will have lost your data due to this third party attack.

We are offering all customers who have lost data a brand new VPS on our new platform. If you are in the position of requiring a new VPS due to your old one not coming back up, please submit a support ticket with the subject ‘New VPS Required’, and including the specification you ordered in the message. We will then start to provision these straight away. We will aim to have all new servers up within 6 hours at the latest, of course providing no new issues occur. If you have your own backups you can then restore these onto the new VPS.

We will also be providing two months free hosting as compensation to customers that have lost data and require a new VPS.

We apologise for any inconvenience caused, and your patience and understanding in these very difficult times.
Regards, VAServ Team.

21:03 vz54uk dead/100% loss
21:05 vz51uk dead/100% loss
21:08 All failed 100% vz16uk / vz22uk / vz28uk / vz32uk / vz38uk / vz41uk / vz57uk / vz59uk / vz66uk
21:11 A batch of new uk vps’s have been delivered to VASERV customers. This does not cover all request, more will be done.
21:29 FsckVPS: We still ask that customers only have one ticket open and only update that ticket. This will help track each clients individual needs. ETA for ticket response time is currently 2 hours.
22:10 Extra Staff called into both locations to help with tickets responses. We currently have about 600 tickets open between our brands. Were still attempting a 2 hour response time for tickets.