1

We are starting a new computer lab and expect to have data that is 50-100 TB of text, images, and maybe videos to process. We will need to run tasks occasionally (10-30% utilization per month), so mostly it will be used for storage. Should we go for a server or cloud computing? Some points to consider (I am not expert in hardware):

Server pros:

  • Last for years
  • Cheaper

Server cons:

  • Maintenance, upgrade problems, and other issues are common given that our expert is always busy

Cloud pros:

  • Easier to use (no hassle)

Cloud cons:

  • Might be expensive to host large data
  • Need to pay for it after 1-2 years (some funds needs to be spent within 1-2 years).

closed as off-topic by Massimo Ortolano, Buffy, Azor Ahai, Dmitry Savostyanov, scaaahu Jun 12 at 2:08

This question appears to be off-topic. The users who voted to close gave these specific reasons:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    Surely this depends primarily on the host institution's rules and resources. – Thomas Jun 11 at 21:37
  • 4
    I suspect people will have an instinct to close this as a boat-computing question, but I urge them to think first; this is a question that many in academia are having to answer at the moment, and IMHO there are some aspects that are not in common with the same question elsewhere - plus, the people making the choices in academia don't tend to have the same expertise as IT infrastructure people elsewhere. – Flyto Jun 11 at 22:17
  • 1
    @Flyto This is a question specific to a research field. It's like me asking here if I should buy an atomic clock for my next experiment or if I better lease it or ask to the lab next to mine to provide the signal. – Massimo Ortolano Jun 11 at 22:28
  • 2
    @MassimoOrtolano is it, though? Many people across many fields face this question. – Flyto Jun 11 at 22:30
  • 3
    @Flyto But the answer is going to be very different based on things like if 100 TB is a lot for you or not that much; or the technical ability of your staff. A group in the CS department might have a very different time getting a cloud going than e.g. a linguistics department branching out into some more technical forms of analysis for the first time. – Azor Ahai Jun 11 at 22:42
2

Local servers are likely to be substantially cheaper and also faster to access the data over LAN rather than the internet for data on that scale (just for a quick count from AWS you're looking at $2400 per month if you ignore the deep-storage options that are not appropriate for data in-use; for the same price you could pay for local servers in a couple months with the same capacity).

Given similar choices, the labs I've worked in have always opted for local storage, but we also utilize cloud options for sharing data both for personal use (i.e., having access at work and home) and between other labs. However, this is only a subset of the data, for example data extracted or summarized from raw data.

2

This isn't an answer - because the correct answer, if there is one, will depend on your circumstances and what exactly you need to do with all that data - but here are some things to consider:

  • If you have your own server, then in addition to the CapEx to buy the system you must budget for somebody's time to maintain it (maybe a group member, if somebody has the requisite knowledge, or maybe somebody from your institution's IT department), electricity (if you pay for that), backups, and so forth. (do not neglect backups - if your main need is storage, this may nearly double the cost)
  • If you use a cloud facility then you have all OpEx and no CapEx - but the operational costs will be substantially more.
  • In my experience looking into this for compute-based HPC tasks, if you can't keep your own system busy then it's often better value to use cloud offerings, during the life of the project. If your main need is storage, this may well not apply.
  • Once the project has finished, if you have bought a server then you have an outdated server that you may, or may not, be able to usefully repurpose. If you have used a cloud system then you have nothing.
  • Consider transfer speeds to/from the storage. If your data and your processing are both in the cloud that's fine, but don't plan on copying tens of terabytes between local and cloud systems on a regular basis.

So far, those are considerations that apply everywhere - not just to academic projects. But here are some additional considerations for research:

  • If the need is mostly for storage and not serious computation (not clear from the question) then up to 100Tb isn't that much storage by today's standards. It's more than a university IT department is going to offer for free, but you might be able to pay them out of project funds and not have to worry further about it. That way you get a resilient system that's looked after by professionals within your institution. They'll probably be buying the same hardware that you'd be buying yourself, so this isn't a cheap option - but it may be a good one.
  • Consider whether, at the end of the project, you will still need to store that 100Tb - or whether the archival need is less. Consider whether, and how, this relates to any funder data archiving policy, and/or any instutional archive facilities.
  • Some funders like to be able to take pictures with a thing that they bought (though you're probably talking about bigger computers than this before that kicks in). Similarly, some funding streams will only cover CapEx and some will only cover OpEx, so this may be the deciding factor.

There are probably some other factors that I haven't thought of - if you've thought of one - especially for the second section - that you don't think is worthy of its own answer, leave a comment and I'll edit it in.

1

I would first look to see if there are HPC (high-performance computing) resources associated with your university (or funding agency, or wherever you're eligible to apply for an allocation). Not all of them are for pure number crunching - some are very suited (even designed) for data intensive tasks. Usually that doesn't give you long term storage, but getting an allocation with temporary scratch storage on the order of 1-10 terabytes is quite feasible. And you could probably pay a bit to extend the storage, if needed. Should still be cheaper than setting up your own cloud solution, and less work than a custom server.

Not the answer you're looking for? Browse other questions tagged or ask your own question.