Today Bitomat, Poland-based third largest Bitcoin exchange, announced that as of July 26 they lost access to their wallet.dat file and thus the exchange’s entire fund base has become inaccessible. The Bitomat service is now up for sale for 17,000 Bitcoins to help recover the cost of the missing coins and return them to their owners. At a current conservative market estimate of near $13 per BTC that would put the sale price of the exchange as exceeding $221k.

According to the statement published by Bitomat, it appears that they were using Amazon Web Service’s Elastic Cloud Computing network and held all of their information, including the wallet file for the exchanges bitcoins in a virtual machine. Here’s a portion of an unofficial translation of the statement,

On 26 July 2011 at about 11:00 PM, I noticed that bitcoin server was out of resources and I had to increase RAM. As a result of this operation, the virtual machine was deleted and all data lost, including bitcoin wallet and its backups.

I have established that data was lost because settings of the virtual machine were changed, although I didn’t change them myself. Amazon Web Services Company, which hosts our servers, says that the cleared machine has been set up to be irretrievably destroyed (including the data on the disks) at the shutdown.

I’m still trying to establish who has changed the settings and whether I will be able to recover the lost data. Unfortunately cooperation with Amazon Web Services is very difficult. As soon as I realized that my virtual machine was lost I have ordered AWS premium support, talked to the manager and asked for securing of the disk data. So far, without success.

As most reporting on Bitomat’s losses are in German or Polish currently, it’s hard to get a clear picture of exactly what happened. However, from the translation above it’s apparent that they were using Amazon Web Services Elastic Cloud Computing to host virtual machines and that the instances that ran their exchange had been reset.

AWS contain numerous warnings to users that the virtual machines running in the EC2 cloud make use of ephemeral instances; meaning that if an instance is taken offline all the data stored within can be lost permanently. Taken from the statement, it appears that Bitomat happened to be storing backups and current state of their wallet in an EC2 virtual machine so it’s possible that they have little change of recovering the old funds from the wallet.

A case of an entrepreneur using the wrong technical solution

Ordinarily, cloud-computing and -storage for this sort of operation would not have led to this outcome; however, the EC2 was not designed for the sort of persistence that Bitomat was using it for. As a system, the EC2 is primarily designed for high-powered, distributed computing processing that provides short-term storage needed by number crunchers. For persistent and durable storage, AWS offers the Amazon S3 (Simple Storage Service).

By keeping both the wallet and the backups on the same ephemeral virtual instance, this sort of failure is almost inevitable.

Right now, it appears that Bitomat is up and running again with a fresh wallet.dat file.

The proprietor has also mentioned he will be reversing wire transfers that have not be turned into bitcoins yet.

The last time that AWS EC2 crashed and went offline (in large chunks) it did take out websites with it such as Foursquare and Reddit; nevertheless, neither Foursquare nor Reddit used EC2 for storage and didn’t lose any data—they simply couldn’t provide processing time for incoming requests and were severely slowed for the duration. This is the ideal outcome for using a cloud-service, making certain to use each service for its provided nature. A definitive postmortem of that event generated a lot of turmoil over the use of cloud services to render popular web services.

Bitcoin exchanges racking up a history of damages

This event will go down in a long litany of Bitcoin events that have caused massive losses in the BTC economy. It started with a hack that hit MtGox that lead to the actual theft and attempted gaming of the bitcoin economy by the hackers. Then there’s been some drama developing between Tradehill and Dwolla. The market remains fairly stable recently but it also means that bitcoin customers may need to start expecting more out of their exchanges.

In the wake of the hacks that nearly obliterated MtGox, the site has been working towards acting more like a currency exchange with more bank-like security. More exchanges will have to follow suit. MtGox is simply the previous exemplar because they grew out of a hobby-server providing an exchange for Magic: the Gathering collectable trading cards, not exactly the venue that expects to need enhanced security to protect themselves from hackers and scammers.

As Bitcoin becomes more and more popular, currency holders and exchanges will need to pave the way for a future where they recognize they’re trading in an valuable commodity. Otherwise BTC will remain an obscure, albeit interesting, cryptographic currency concept that no big players like PayPal or Walmart want to deal with.