Home > Articles > Cisco Network Technology > General Networking > The Art of the Data Center: Bahnhof - Stockholm, Sweden

The Art of the Data Center: Bahnhof - Stockholm, Sweden

  • Sample Chapter is provided courtesy of Pearson.
  • Date: Oct 9, 2012.

Contents

  1. The Interview

Chapter Description

Doug Alger takes a fascinating look at one of the most amazing data centers in the world in this chapter from his book The Art of the Data Center.
Figure 3-1

Figure 3-1. The above-ground entrance to Bahnhof’s Data Center in Stockholm, Sweden. Embedded 100 ft. (30 m) underground in a former nuclear bunker, the Data Center retains its wartime codename, Pionen White Mountains. Images provided courtesy of Bahnhof.

Essential Details

ORGANIZATION: Bahnhof

LOCATION: stockholm, sweden

ONLINE: september 2008

NOTABLE FEATURES: artificial daylight, greenhouses, waterfalls, and a 687 gallon (2,600 liter) saltwater fishtank. Two Maybach diesel submarine engines used for standby power.

TIME TO DESIGN AND BUILD: 20 months

SIZE: 10,764 sq. ft. (1,000 sq. m) total, with 5,382 sq. ft. (500 sq. m) of hosting space and 2,153 sq. ft. (200 sq. m) for back-of-house systems. remaining space is for office and personnel areas.

IT-USABLE CAPACITY: 800 kW

TIER: not stated

CABINET LOCATIONS: 140

POWER DENSITY: 5.7 kW average per cabinet, no specific maximum

INFRASTRUCTURE DELIVERY: Cooling, structured cabling and electrical are delivered below a 3.3 foot (1 meter) deep raised floor.

STRUCTURAL LOADING: 403 lb. per sq. ft. (2 tons per sq. m)

FIRE SUPPRESSION SYSTEM: novec 1230

The Interview

What made you choose a bunker as an ideal location for a Data Center?

There were several reasons for it, but the primary one I think (is that) we had built five Data Centers and most of them were built in traditional environments—old warehouse redesigns and old office building redesigns—and we wanted to do something different.

We are geologically in a very stable area. Most parts of Sweden are built on granite. The mountain, the stone is solid rock and it’s 2 billion years old. It’s one of possibly the earth’s oldest and most stable from a geological perspective.

There are fortresses and stuff from those days of the Cold War and I looked through most of them. I think I was in 40 different spots. Most of them were too small or scrapped or used for something else, but this facility was still there. We were very lucky to get hold of it.

If you work with computers, you realize that the threat to operational computers is not too much. The physical protection, okay, it must be there of course. It must be there, but now we have a mountain defined originally to (withstand) well not a hydrogen bomb outside the door but at least a hit somewhere in Stockholm and the computers probably survive even if people may not.

The place was very big and it was also located very central in Stockholm. Our competitors, and many Data Centers, are often located way outside the inner city urban area because they can be built in a larger complex outside for a cheaper price. But, you see, many computer consultants they live in the town. They don’t want to travel far away to look at their boxes. It’s nice to have them conveniently close to the city, actually. Of course it’s very hard to find a great, cheap space where you can build a Data Center in this urban city but with one exception and that is these caves because nobody can use them for something else.

Sometimes they have been used as garages, but (at this site) the entrance was too narrow so it was impossible to build something like that. I would say it was a clear benefit that the space was so centrally located and that gave us an advantage against our competitors. Our clients can easily access the computers and after they have accessed it they are in the main city area.

Figure 3-2

Figure 3-2. The Bahnhof Data Center’s Network Operations Center (NOC). Artificial daylight, greenhouses, and even a saltwater aquarium have been installed to avoid the impersonal feel of many conventional Data Centers.

The second reason: of course, the physical protection was also great. As you know, when you work with a computer center, it might not be of that huge importance. Most errors in my work are basically caused by humans. Human error—that is the most frequent. Mistakes in configurations and whatever can make your environment go down. Not from some kind of big trouble with the physical protection.

But that doesn’t matter because the clients they like what they see. From a marketing perspective, they’ll of course appreciate that they feel secure. Even if I know that humans are more cause of problems than the physical location. They appreciate the solid feeling of the rock. The solid feeling of the rock gives us an advantage compared to a conventional center.

The third thing was a combination. It was fun to do it. It is like playing. The fun-ness of it also brought us an advantage in marketing. I mean, I speak to you now and we have had a lot of magazines and footage from different media channels covering it. And of course that has given us invaluable access from a marketing perspective. If I had to spend the amount of money on advertising it would have been impossible. Now we have all of this, if not for free, it has given us an advantage from the marketing perspective.

The style we chose, that was about the fun-ness of it. It’s a French architect who has been responsible for the design, but he has been working very close to us and we have discussed ideas.

The great inspiration has been one-part James Bond and one part Star Trek or science fiction movies and stuff like that. That has been an inspiration source to do it like this. The only thing that is missing, I was considering a white cat with long hair, like the character in James Bond, that Blofeld guy (Ernst Stavro Blofeld), and sitting in our conference room in glass and have this cat. We have plants, but no pets right now.

Figure 3-3

Figure 3-3. A circular, glass-walled conference room overlooks the Bahnhof Data Center’s server rows.

This room definitely makes an impression upon people when they see it. Did you have a sense when you were designing the Data Center that it would be so attention-grabbing?

Absolutely. There is something magical about caves. I don’t know what it is but people are sucked in to the facility and immediately like it. Even when we were at a very early stage.

We defined it completely from the beginning.

We have blown out 4,000 cu. m (141,300 cu. ft.) of stone additionally, on what it was before, with dynamite. We were building it for almost for 2 years. The neighbors in the area were not so happy from time to time from this dynamite. Once accidentally there was an explosion and the coverage from the front door was not enough so a small piece of stone penetrated a car outside the entrance. Luckily there were no people there and nobody was harmed.

When we had it very early people were very interested in it and they wanted to come and see it and looked. Right now we have delegations from companies and universities and interesting parts. And our policy has always been to be very open about it. We invite people and we have meetings and visit the facilities—not where we have the servers, in that space, but in the other parts of the facility where you can see it very well.

How long overall did it take to design and build the Data Center?

Maybe two and a half years from the start. We hadn’t built a Data Center in solid rock before and this was so different. It depends upon where you build it. If it’s in a warehouse it takes a certain amount of time but this was so special. It took us about twice as long as we foresee from the start.

This was a success. It has been a commercial success and it’s a success from a design point (of view), from all angles, actually. Even so, I’m not sure if I had another opportunity to build a similar Data Center again I don’t know if I would do it because it was so tiresome to remove all this stone, and it was a risky project. It could have gone wrong. There could have been problems with water from the groundwater. There could have been different types of problems. It is cool and it has been a success both technically and commercially but it was a lot of work to do it.

I remember when I presented it (the proposal) to the bank, when we had some investment loans to carry some of the initial investment, and I told them that we were going to build a James Bond fortress in the mountain. This is long after the dot-com time. It might have sounded like a dot-com project.

Well, it turned out good and we had investments that are not huge. We kept a very tight budget so it was built with very little money, actually. We put in mostly our own work and found solutions ourselves to do it.

Figure 3-4

Figure 3-4. Solid rock surrounds the server cabinets within the Bahnhof Data Center.

Obviously, having to blast through solid rock to expand your usable space is a major challenge. Were there other challenges specific to this site or the fact that you were building it underground?

Each problem was very unique. If you have built several Data Centers in conventional spaces you know basically what problems there are and what you have to do. But with this every day was a new one.

I was worried that big huge blocks of stone might be loose and just fall down, but it was not the case. It was really solid. We had some at the entrance we modified something and when we blew up the space after that we added cement just to fixate some of the rocks at the entrance.

I was talking about that before, it was built in the urban city area. People were sleeping and living very close by. It was not a business-to-business area. There was a church above us. I was worrying that (someone from) the church might call me and say ‘Well, we have to tell you that the church has fallen to pieces’ and that would not be so good. That was my nightmare, but it didn’t happen.

We actually had measuring equipment placed on the buildings outside just to see if there were any troubles caused by the dynamite, but it turned out alright and we didn’t encounter any such problems.

The area where the Data Center is located, about 30 meters (98.4 feet) above us, we have these old, 18th-century wooden houses. So it’s a big contrast, we have this fortress in the mountain and then you have these small wooden houses. It’s a cultural park, a preserved area for older, traditional Swedish houses and then we have this big church also. We have this contrast with the high high tech, science fiction facility and the old environment outside.

I imagine it was difficult just bringing some of the traditional Data Center infrastructure technology below ground, even once you had the space cleared out the way you wanted it.

Yes. We lowered the floor 2 meters (6.6 feet) almost, to make the room have better airflow. I would say the actual computers they are based on traditional level floor cooling where they blow up cooling from the floor and into the cabinets. I know there are different technologies today, but our clients are mostly co-location customers. They come with different boxes of all types and they want access to all types.

It’s very hard to build individual coolness in individual cabinets unless you know exactly what type of computers or configurations people are using. When you’re doing open co-location space it must be very flexible. We have clients even with old power models, God forbid, but that is the case. And the client is always right. Then it has shown the most convenient way to do it is to have this level floor cooling model.

In all other aspects it’s a conventional Data Center from a technology perspective. We have great redundancy. The network is coming in from three different places. The electrical power is also coming in from different places. The unique stuff is the atmosphere and the protection it gets from the mountain.

At the early stage we discussed the use of some kind of geothermal cooling: you drill into the mountain and bring up coolness from the ground. This turned out to be insufficient, we have too much energy. We investigated that but I was told we would have had to drill under several hundreds of homes and it would have been of effect for a limited time span. Sooner or later we would eventually warm up the entire mountain and then (the geothermal cooling) would have no effect. So we decided to use the outside air cooling instead. One advantage in Sweden is that the climate is colder, so we can very often use the outdoor temperatures in the cooling system. We use conventional cooling also but using free cooling from the outside air.

Figure 3-5

Figure 3-5. The underground Data Center incorporates outside air cooling rather than geothermal cooling, to avoid warming the mountain that surrounds it.

Did you have any particular challenges around ventilation and using outside air, with the Data Center being embedded in a mountain rather than in a conventional space?

There was no problem in that. It was built to have air in it in the beginning because it was built for people. There is a ventilation shaft going up, on top of the mountain where we bring in the ventilation for breathing. It’s very clean air. It’s fresh air in most parts of Stockholm, but it is exceptionally good because it’s very high up also. We have very good air inside there.

Right now there are 15 people working there. It’s our NOC, basically, the network operations center for our Internet telecom operator. We have been there in an operational sense since it opened in 2008. We have no problems.

There are plants there, also. We had to bring some organic feel into it. If you are forcing people to sit in the mountain you must have something that humans like. And that is plants and we have a big fish tank, about 2,500 liters (660.4 gallons) I believe, with saltwater fish. We tried to simulate daylight, stuff like that. It’s very much built for people.

Did you know from the beginning that you wanted to have those sort of features in a facility or did that come later?

Definitely. That was initially planned. They were in the budget from the beginning. It was a strategic decision very early. We decided that this should be a nice James Bond fortress and if you have a James Bond fortress you should have a fish tank.

We originally were planning to have piranha—you know, the little fishes that eat meat. But I was told that they are so boring. They just stand still and are not so colorful. So we decided for saltwater fish and colorful instead. And, the plants, I think it was a science fiction movie Silent Running, something from the early ’70s, where we had these space stations with greenhouses floating around in outer space. This was the original idea. Everything was defined with this in mind. It was not added on later. It was the idea to have all this from the beginning.

Figure 3-6

Figure 3-6. Artificial waterfalls within the Bahnhof Data Center.

It’s impossible to argue with the logic that if you’re going to have a James Bond villain lair you should try to include fish tanks with piranha in them. That’s certainly consistent. You mentioned the raised floor system for the delivery of cooling. Is the Data Center’s power and cabling infrastructure delivered under the floor as well?

Yes, that is under the floor. That is why we lowered the floor so much, to give us additional space. I know that you can have it above, but aesthetically we didn’t want to have too much stuff (visible). It should be clean and look science fiction-properly. That is why we decided to have them under the floor. It is a higher (deeper) space than normally so this gives good airflow flowing in. I know sometimes you get problems with the cables under (a raised floor) because they can hinder the airflow, but this didn’t happen. The raised floor is definitely higher than in our conventional Data Centers.

If you could go back and design the facility all over again, is there anything that you would do differently?

Maybe blow out more space. Because now we have all of the cabinets are almost gone (occupied) now. We should have made it bigger. That was the mistake. It was a matter of cost, also.

Second, maybe even add even more (features). I designed with the plants and stuff like that, but we scrapped other ideas. With the ventilation shaft there was a possibility to build an elevator, and I was considering to have something like a Batman elevator (with Bat-poles) going in to the cave from the top of the mountain, but it was too expensive. Given the chance again I would definitely add the Batman elevator to the facility.

We were also considering a small railway from the entrance down to the center, but it was too complicated. Given the chance again I would add something more like that. And my recommendation for guys building computer centers is (to understand) that people like this kind of stuff. Even if you don’t have a cave you should definitely consider adding stuff that gives some kind of human touch or a feeling or atmosphere for the facility because computer centers can be so boring.

Many of our customers, it’s included in our offering, they can bring their own clients and can have presentations (at the site). It’s a great facility to have a presentation if you have some kind of new website or project to present. The clients appreciate having meetings in this kind of environment. That enhances their experience and gives them great value for placing their servers and applications here.

Figure 3-7

Figure 3-7. Two Maybach diesel submarine engines, complete with accompanying sound-horn, provide standby power for the Data Center.

You’ve anticipated my next question somewhat. Not everyone would entertain the idea of putting a greenhouse or these other features in their Data Center. For the amenities that you have installed, do you think these are universal in that anyone who puts them in to their Data Center would see a benefit or are they useful at your site solely because you decided to have a theme?

Let’s face it. Data Centers have been built by technicians for technicians but sometimes we have forgotten about the human touch and to give some style on something. I think you have to add that.

I don’t say that you should have to build a fortress in this special style in a mountain that we have done. But if you have a conventional center you can always enhance your experience by playing around with stuff and doing it differently. Something that gives it a unique feeling. That is often neglected. People spend a lot of money on traditional marketing, but just by giving these small items you can bring a lot of extra value. And many of the clients appreciate it, too. From that point of view it’s definitely worth it to invest and give that. Especially if you want people to work in the Data Center. People are more productive and happy if their environments are nice than if you are working in a boring space.

Do you have plans for any future Data Center buildouts—for any other Bond villain lairs if you will?

Yes. We are working on a concept with modular Data Centers. The idea sprang from that fact that if you build in a mountain you realize that the mountain it is very hard to expand in space. Once the space is finished you have to do something else. So, I would like to build Data Centers in modules where you can have module after module after module when you need more space.

With those modules, are you also going to put in any interesting features?

Absolutely. Absolutely. I’m considering building a space theme. Imagine a space station on Mars or something like that, with modular tents and very scaled design. It should be defined.

If you consider the Swedish products like the Absolut vodka bottle. We can always say that the bottle is a bottle. But if you add some design and add some concept with it it enhances the experience. It’s the same. If I say ‘We are going to build modules’ people will believe it’s containers, but it’s not containers. They will be designed differently and they will for sure give the fun factor we are looking for.

I have gotten a lot of publicity for this project. A mountain is a mountain, so maybe that is hard to top in terms of the atmosphere. But we will give our best to do it.

Any final advice that you would offer to others who work on Data Center design projects in the future?

I think you should give the human factor a great thought. Computer centers should be built for humans and for their clients. Most often you focus a lot on the technological aspects but you forget about the humans who are going to work there. Also, giving this design you add an experience for the client. That is very often forgotten in these type of projects. Bring a human touch to the computer center, that’s my advice.

Figure 3-8

Figure 3-8. The Bahnhof Data Center’s electrical room.

Figure 3-9

Figure 3-9. The reception area at the Bahnhof Data Center.

Figure 3-10

Figure 3-10. Greenery is planted in the Network Operations Center (NOC) to make the environment more comfortable for employees to work in.

Figure 3-11

Figure 3-11. The Bahnhof Data Center’s fire suppression system features a gaseous suppressant.

Cisco Press Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from Cisco Press and its family of brands. I can unsubscribe at any time.

Overview

Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about Cisco Press products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites; develop new products and services; conduct educational research; and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@ciscopress.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security

Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children

This site is not directed to children under the age of 13.

Marketing

Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out

Users can always make an informed choice as to whether they should proceed with certain services offered by Cisco Press. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.ciscopress.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links

This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020