I would like to use an supercomputer to run my big data project to test it speed. I google the service & see that only this is available for free but it closed last year? Could anybody with experience in using supercomputer know to get access to this kind of hardware?

  • An hour wall clock time or an hour CPU time? The former will not be very useful, as just the time to configure your software to run on a supercomputer will likely exceed this time. – gerrit Jul 2 '18 at 10:20
  • i mean an hour wall clock time. I'm not plan to use it on long term. I try to solve a problem that how long can it take for me to search & filter a database with 5 trillions value in it. Obviously normal computer can't do this. I read a lots about china supercomputer can do these kind of things in seconds but i don't think i need that much power – Huang Lee Jul 2 '18 at 10:31
  • 3
    It does not sound like a "supercomputer" will help you here. The time it takes to copy several terabytes of data to any other machine will be slower than searching for values in them locally. – koalo Jul 2 '18 at 10:48
  • oh, i forgot about this – Huang Lee Jul 2 '18 at 10:52
  • 4
    Important thing about supercomputing/HPC: Figuring out what the bottleneck is. If your application is I/O-limited (which it sounds like it might be), a "generic" computing cluster or supercomputer probably won't be much help. Also note - since it sounds as though you may not have realised this - that you can't just take something you wrote for a home PC and expect it to go hundreds of times faster on HPC hardware. Most HPCs have similar, or slightly slower, CPUs, memory, etc., than your office PC - they just have lots of it. So you need to put effort into parallelizing. – Flyto Jul 2 '18 at 12:28

The answer will depend on where you are, who your funding is from, and what you mean by "a supercomputer".

Various countries, research councils, etc., let you apply for computer time if you are eligible. Some universities have their own in-house systems too.

But also, the major cloud providers - Amazon, Microsoft, Google (?) - will let you rent computing resources. This may be suitable for your needs.

  • My local university never has this kind of supercomputer hardware which is why i'm seeking it online. On average, how much it cost to rent supercomputer for an hour? – Huang Lee Jul 2 '18 at 10:33
  • 3
    @HuangLee Renting it for an hour is rare because most real problems can't be configured and solved in an hour. – gerrit Jul 2 '18 at 10:40
  • lol, this is why i try to find a free or trial service =]] – Huang Lee Jul 2 '18 at 10:49
  • 4
    Depending on the size needed, an hour on Amazon EC2 would probably cost less than a cup of coffee. In fact you might be able to do some work on their "free tier" (I forget exactly what's included). There's considerably more than an hour's learning curve associated. But as noted in comments to the question, if you are dealing with a very large dataset that isn't already in the cloud, the upload time might be more than the processing... – Flyto Jul 2 '18 at 12:25

What is your location and funding availability? Typically, one can get some time share on national scientific supercomputers through grants. You might use local clusters of your lab or university for testing and slower, but cheaper and easier available resource.

An alternative are clouds, e.g. the one from Amazon. It is quite cheap in processing time, as compared to commercial offers. It is also readily available. So, if you want a quick test and have money (in contrast to political influence or grants), it might be the faster way.

But generally, there are grids, there are cloud services (that are the same, but somewhat different), and there are the classical supercomputers (still grids, but own hardware, faster interconnections, etc.) While the principles are quite similar, you might need person-months to tune your application to the specific platform. Assuming, you want to squeeze maximal performance, of course.

A possible alternative might be GPU computing. If your task maps well to grid, it might map well to CUDA or OpenCL. Again, there are months of development to get it right, but probably lower entrance costs.

  • how fast GPU programming possibily can do when you use it on search algorithm? Like can it search about billions value in seconds? Can you be more specific, i heard about this but never have any expreince – Huang Lee Jul 2 '18 at 10:48
  • @HuangLee That entirely depends on the actual algorithm. Wikipedia might be a good introduction. – Konrad Rudolph Jul 2 '18 at 12:54
  • @HuangLee To expand on Konrad's answer... The processors in supercomputers do not run any faster than your PC. The difference is purely having hundreds or thousands of processor cores available. So you need an algorithm which can easily be split over cores, otherwise it won't run any faster than your PC. – Graham Jul 2 '18 at 13:02
  • @HuangLee What are you searching for? How is your domain structured? Are there massively parallel implementations out there? How much interaction and communication between the nodes is absolutely necessary? Is memory locality given? Basically, you can put a separate PhD student on such a task – if there are no clear and evident solutions yet. – Oleg Lobachev Jul 2 '18 at 13:20
  • @ Oleg Lobachev um, actually, its' not that complicated :| – Huang Lee Jul 2 '18 at 16:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.