54

I'm currently working as a PhD student.

I am not working on a tech-oriented PhD. However, my project requires a lot of programming. This involves both working on a computation cluster, servers, and working with a variety of software. These are provided and maintained by our faculty IT support.

Now on the reason why I post: the first problems started when working on our server, maintained by our IT staff. I required some software essential to do my work, but because I have no administrator rights, I cannot install what I need. I asked IT very politely (I'm the new one after all), and they let me know that they would handle it as soon as possible.

Two weeks later, I still did not have access to the software I desperately needed to do my work. I had to ask my promotor (luckily a huge help in this matter) to ask IT, and finally the software was installed.

The following cycle continued until now:

  1. I notice that software is missing / outdated / not working properly
  2. I ask IT to fix the issue, because no other options are available
  3. IT ignores my requests, leading to me having difficulties to meet deadlines
  4. Someone with better credentials has to pressure IT into fixing the issue

Like I mentioned earlier, I absolutely love programming and computers in general, so I can fix almost all issues myself. However, my hands are tied because I have no administrator rights, and IT goes to great lengths to avoid that a non-IT person does something / installs something on one of the computers...

Does some here have similar experiences, and do you know how to smoothen this out? I obviously don't want to cause a ruckus in our department, but I also believe it to be unfair that we are blocked from doing our job. I'm not the only employee who was bottlenecked by IT, so I guess all suggestions are welcome!

Improve this question
20
  • 19
    Is there any chance of you installing your required software in your home directory rather than as root?
    – thosphor
    Mar 21, 2019 at 13:58
  • 76
    Having been on the IT department side of this, and in case you're curious, the most likely reason for their behavior is that they have a lot of more urgent work that they are catching up on, and only once someone higher up the ladder makes your request urgent does it get ahead of all the other things. To the detriment of the work on other things. So for what it's worth, they are probably not ignoring you for any personal reason, they are just scrambling to keep the servers running.
    – Todd Wilcox
    Mar 21, 2019 at 14:21
  • 12
    In the meantime, you could try to have things working in your computer, with a smaller dataset. This way, you know exactly what you need to make your thing work and you can go to your professor to ask them to ask IT to install all you need at once, instead of doing the cycle you describe.
    – user4052054
    Mar 21, 2019 at 14:25
  • 12
    Does anybody not have this experience? This is what IT departments do. It's the easiest way to meet their objective of "reducing expenditure."
    – alephzero
    Mar 21, 2019 at 18:32
  • 15
    Almost every IT department is under-staffed, because management treads it like facility management: you have to have it, but it has no business value. They do not notice the problems, because their requests are always fulfilled immediately.
    – user66337
    Mar 22, 2019 at 8:03

13 Answers 13

Reset to default
50

This is a partial solution I've used myself, and it assumes a few things:

  • You access your clusters remotely through a connection (like other answers pointed, cloud servers or remote), so where you work isn't exactly the bottleneck.
  • The software you use isn't OS dependent or has any special licenses
  • You do not have a ton of things hooked up from serial ports to measuring equipment and whatnot.

One thing I've done in this situation is, when you finally grab a ticket from IT, ask them to install a virtual machine like VirtualBox and have them enable virtualization in the BIOS. Most processors allow this nowadays. Make sure they install it properly or they will ghost again.

This basically allows you to install any Linux distribution you want (you need Licenses for Windows images), so it is an operating system where you have all admin rights. This completely removes the IT middleman if your case is just coding and clustering the data crunching since you don't need their permission to do things in the VM, and also shouldn't impact performance significantly.

If IT is worth half a dime, they will appreciate the solution, since:

  • It takes gargantuan effort to compromise the Host Machine through a VM if you are not malicious
  • Resulting lack of tickets from you needing something updated/installed.

Worst case scenario you can still alternate between both Operating Systems, and do small things in the VM with your admin rights.

Improve this answer
8
  • 37
    I'm not sure about university IT departments, but in other areas, a user that does not have admin rights to a server cannot have a VM on that server with admin rights on the VM either. In other words, I've never worked in an IT department where this request would be granted. Something that would work in my experience is getting approval from your department or program for money for a virtual machine on a cloud service like AWS or Azure.
    – Todd Wilcox
    Mar 21, 2019 at 14:18
  • 3
    @Todd: In principle, anything a user in as an admin guest OS can also be done as a user host OS. Depending on how the IT department models threats, this sort of request might be granted. Where things get problematic is with VM-based security systems. E.g., Windows 10 environment has a feature called Device/Credential Guard. This security feature tends to prevent VM software from functioning. Even if the IT department is willing to toggle the BIOS virtualization settings, they'll probably be far less willing to disable Device/Credential Guard.
    – Brian
    Mar 21, 2019 at 14:36
  • 2
    If the only time you need admin rights is the installation then a singularity container should do the trick. One of these things; sylabs.io/docs afak they are cli only, but otherwise they are basically a virtual box with better pass through. You can set it up as admin on your own machine then move the image to the cluster and run it there as a normal user.
    – Clumsy cat
    Mar 21, 2019 at 18:16
  • 6
    @Todd Wilcox: As of a couple years ago, you could get a free student account on AWS. Depending on what the OP's computation needs are, that might serve.
    – jamesqf
    Mar 22, 2019 at 5:43
  • 1
    Considering that a great many "servers" these days are virtualized to begin with, it's not that simple to ask for a VM any more. Nested virtualization has a whole lot of challenges. Also I don't understand what tools people are running that require admin permissions. I've worked on tons of clusters where I simply installed all my programs for the my user.
    – Voo
    Mar 22, 2019 at 18:18
24

I think this is a fairly common occurrence in many companies. There is a big disconnect between the researchers trying to do their work and the IT staff who may be under-resourced, under-trained or limited by administrative policy. In my experience there is often lots of worry about researchers installing software without fully understanding the security implications. In short, I would not expect you to be able to change this policy and it may well be in place for sound reasons.

One solution that has been suggested to me is that academics can apply for time using different cloud compute solutions (e.g. Google compute https://cloud.google.com/edu/, Microsoft Azure https://azure.microsoft.com/en-us/education/, any many more). If successful you will be given compute time in a virtual machine where you will have appropriate administrative rights to install any software you want.

Improve this answer
3
  • Thank you for your reply. I understand your point, a lot of people just install random software without thinking through what it does to their computer (or the network the computer is connected to). The cloud is indeed a solution I did not immediately think of (I thought it was always a paying service, but I see that they also award grants for using the cloud in academic settings). I will definitely look into it!
    – unit_with_a_soul
    Mar 21, 2019 at 12:36
  • 13
    This might come in very handy but be aware that there might be data security implications. Uploading research data to anywhere outside of my institution's network, especially to US-based servers, could very much be a punishable offense where I work. Of course this might depend on the project, type of data etc and it might well go unnoticed, but it's worth mentioning I think.
    – smcs
    Mar 22, 2019 at 15:05
  • 1
    @speedymcs Absolutely. OP needs to get permission from the relevant people, in writing. One web app I maintain at my old university has to be restricted to one university building, and faculty who want to connect outside the building have to use the university's VPN. It's locked down even to me (VPN + 2-factor auth) as the developer. Universities take data exposure very seriously, so make sure you can actually use the data and protect it in the right way!
    – Chris Cirefice
    Mar 22, 2019 at 16:25
20

We had this issue with some Silicon Graphics machines - we removed them from the IT department's control, with the support of the supervisors whose budgets had paid for them, and then controlled everything ourselves.

It went much better; we had more uptime and fewer issues than any other group / team on campus.

We learnt a lot about compiling & installing etc.

Not sure if you can go that route.

Note, the software we were using was CFD and the IT dept had no experience of what was needed.

Improve this answer
8
  • Thank you for your response. Well, the problem is indeed that it is not possible for us to separate ourselves from IT using budget. The infrastructure we are working on belongs to the university, and the maintenance is a job for department IT. We indeed try to do as much as possible when it comes to the software we use (and this indeed works quite well), but being completely free from IT is really difficult I think.
    – unit_with_a_soul
    Mar 21, 2019 at 12:31
  • 11
    How long ago was this? Today, I'd expect the response to be that the IT support team can't afford for you to be the ones who learn a lot about what happens when you get hacked. Also, nothing in the question suggests that there's money available for duplicating facilities provided by the department. (The asker is using central resources, not resources that belong to the group but which are centrally managed.)
    – David Richerby
    Mar 21, 2019 at 19:10
  • @DavidRicherby They were on an internal network - no external hacking possible.... Funding was by particular research projects not "central" .... Unless you can hack via the power cord...
    – Solar Mike
    Mar 21, 2019 at 19:24
  • 2
    @DavidRicherby It's extremely rare in the US to not have group computers self managed, where the only thing IT requires is current antimalware. They're going to have to adapt quick: had several cases recently where instead of using a university HPC cluster, we just went to Amazon AWS or Microsoft Azure since there you install anything you want on your VM, get the right amount of RAM, cores, storage and GPU, and don't have to wait in line.
    – user71659
    Mar 21, 2019 at 22:54
  • 1
    @SolarMike An air-gapped internal network also means no network-based software updates (and burning a CD to run an update is a pain), installing extra software and the prerequisites for the software is difficult, getting to a login prompt is usually hard. Email and network shares are out of the question. Unless you're running FISMA-high data, I would not recommend going in this route.
    – doneal24
    Mar 22, 2019 at 19:03
13

At many universities, you can actually go through a (rather secret) bureaucratic process that allows you to install your own software on the university computer, even though this is not the default option.

If this option exists, it will likely include signing forms that may limit (or even eliminate) future IT support for that computer. Basically, you are agreeing to be your own IT in exchange for flexibility. However, if you are constantly downloading software it is often worth it.

The University doesn't want to encourage this option, because it can be a perceived security risk, so you will have to directly ask administrators in your department and IT, if this option is available, as it likely won't be published online anywhere. And you may have to assertively push for it.

This process was available to me at two of the three universities I've been based at over the years (in the USA and Australia). The one that didn't allow it actually theoretically did, but the form also said that you would no longer be able to connect to the Universities wired internet network, which wasn't worth giving up for me. Read the forms very carefully before you go down this route to make sure you are comfortable with the tradeoff you are making.

Improve this answer
1
  • 1
    Upvoted and agreed. I was at my current academic position for about 10 years before I accidentally discovered (arising from similar complaint to my chair) that we could request having my office computer be "unlocked" so I could install my own software. Meanwhile, my chair was surprised that this hadn't been done for me years earlier (CS department here).
    – Daniel R. Collins
    Mar 23, 2019 at 8:36
11

Whilst others have focused on giving you some technical advice on alternatives to using your institution's infrastructure, I want to note that your problems are not uncommon. In my practice I opted to installing the software myself (a lot of software packets I used do not need root privileges for installation, make sure that admin access for yours is really a must) or asked supervisor for help.

If you are on good terms with your supervisor, you can always ask him for help, going from as little as cc-ing him in software installation emails that you send to your IT department to directly asking him for help when those request emails are ignored. In many organizations it's not easy for junior members to be listened to and that's just something you have to get over with.

Improve this answer
4
  • Thank your for answering. Yes, I execute several applications as 'non elevated' user because that will give me what I need in most programs. My supervisor is indeed a big help and, for now, asking the supervisor is the best workaround. Yes, I understand that this is not a rare problem, especially for new employees like myself. However, what annoys me is that the things I ask are really not that difficult. I'm usually merely asking to install a certain software update or library that could come in handy, which is not something that takes weeks to complete.
    – unit_with_a_soul
    Mar 21, 2019 at 12:45
  • 6
    Upvoting for the second part here - always CC someone with more leverage whenever you report issues. Just that frequently moves it up in priority a surprising amount.
    – user2699
    Mar 21, 2019 at 14:03
  • 1
    Will chime in that as an admin (non-IT), receiving a cc with the PI is essentially "failing at your job". It is the form of accountability that staff relies on. So as much as we groan when we are at fault, it's absolutely the go-to for reminding other staff that we aren't doing this for our health--other, more important people care about this too. We have a million competing priorities, so unfortunately the squeaky wheel gets the oil... running from emergency to emergency...
    – yourfriendlyresearchadmin
    Mar 22, 2019 at 5:14
  • 1
    Upvoting for the second part here. On the one hand, it is your advisor's job to make sure you have the IT resources you need. On the other hand, your advisor has actual leverage with the IT admins; you don't. Always CC your advisor, and ask your advisor to follow up immediately with "I concur with / approve this request."
    – JeffE
    Mar 22, 2019 at 9:02
7

Since you're not in a "tech-oriented PhD", I'm guessing the cluster is 10 machines or less? If so, consider setting up your own cluster on any of the big cloud providers. AWS and Azure come to mind, but Google and IBM are still trying to hang on in those markets.

Thoughts on pricing:

  • You don't have to have the cluster up all the time, so you don't have to pay for 24hrs/day for a full month. You would have to figure out how to save/load the data and install some programs quickly (docker?), if you opt for shutting down some or all servers.
  • If you have fellow students or professors who are in the same boat, consider sharing your cluster and its expenses.
  • Depending on your research, you might (eventually) be able to get special pricing from some vendors.
  • On AWS, reserved instances are cheaper than spot instances.

Look at how much your time is worth, and how much these delays are impacting your research. If IT delays cost you 1 week of delays every month, then it's probably worth setting up things yourself. (Remember to back things up, and test your backup strategy. Put your code and data in github)

And lastly, are you using their ticketing system for your requests, or just asking them verbally? If there's no ticket, it'll surely be forgotten 5 minutes after you spoke to them.

Improve this answer
2
  • 2
    This answer may apply, but I don't see where you take the logical step from “not-tech-oriented” to “probably cluster <10 nodes”. In fact, to me, needing a cluster at all while not doing tech actually suggest more likely that lots of nodes are involved. Science or data science applications tend to be really computation intensive. Furthermore, many of these applications are quite latency-critical and only perform properly when run on an actual physical cluster with native MPI support and a suitable file system like Lustre.
    – leftaroundabout
    Mar 23, 2019 at 0:02
  • Doesn't AWS EMR allow for there to be clustering on demand in this case?
    – lucasgcb
    Mar 23, 2019 at 12:01
7

You should also be managing your requests.

I mean: track when you raised the request, ask for news after a reasonable period of time (two working days), repeat that max three times. In the first follow-up you can make it clear why this is important to you. For IT staff it might sound like 'I want this tool installed', but if you detail the deadline you have and why this ticket is crucial for your work, they might understand. After the third time, get your supervisor involved. Ask him/her to escalate the request in a conversation with an IT service manager.

Don't be happy with simply solving the ticket, try to see if IT can communicate what time of service level agreement they aim to respect. Maybe installing something requires internal permissions (release management) which, for instance, makes it impossible to install stuff on a Friday, or an afternoon, or if you don't have a rollback (meaning you have to backup the entire server setup before touching anything), etc.

Managing the request also means making sure your requirements are superclear. If you think you might need a package ask them to install it. Run these installations yourself in a VM that is identical to the production setup and provide them with the commands to actually install the tools. All these things you can manage.

At all times, and I must stress that, be nice. Just be nice. Unless you have reasons to suspect someone is trolling you, you have to give them the full credit for having to tackle many such requests at the same time. Being nice will get you further on the long run. It will also show to your supervisor that you are capable of creatively handling difficult situations.

Improve this answer
3
  • Tagging onto this answer: When following up, use multiple modes if possible. For example, start with the ticketing system, but do the first follow up by phone. Being nice and tech-savvy on the phone can work wonders with some IT departments.
    – krubo
    Mar 22, 2019 at 18:25
  • 1
    As an experienced computer tech, I'd have to add that the IT dept. might already have an agreed upon Service Level Agreement (SLA) as to how soon tickets need to be addressed. Some can be as long as a week or to. The OP stating they need to get their software in only 2 days may require extra steps, signatures, or could just be flat out denied due to this prior SLA. I'd suggest the OP finds out what the Standard Operating Procedure (SOP) is before stating any time limits. And being nice is always a good way to stay off the bad side of the IT dept., cause that's a bad side to be on.
    – computercarguy
    Mar 22, 2019 at 20:49
  • 1
    And you need to remember that the SLA is based on the worst-case workload that IT can reasonably expect. In my organization, we promise through an SLA that a reasonable VM will be created within two weeks. In practice, the timeframe is generally within two day and that's only because we have to depend on two other outside groups for part of the process. Also, if installation of your software involves installing multiple cutting-edge packages that affect other users, understand that IT has to support everybody, not just your research needs.
    – doneal24
    Mar 24, 2019 at 20:08
4

Adding this since nobody's mentioned it yet: You could ask your supervisor about having a meeting between you, your supervisor and someone from IT. Maybe even invite them to lunch if it's appropriate to your institution and situation. In case they're overwhelmed by tickets and/or they feel like your tickets come in from a faceless ticket-generator, it could help to meet the real researcher behind them. And you can meet the real person handling the tickets, and see if there's any way you can make your tickets more straightforward for them.

Improve this answer
1

Talk to your supervisor ("promotor"?) about the issue, and suggest that you should get a server that you manage. Depending on circumstances, you can agree on a physical server, an upgrade to you workstation / laptop to make it usable as a server, or even a server instance which is completely external to your institution, like Amazon AWS.

If your supervisor insists that you must use the existing IT infrastructure, send your support requests to them the next day after contacting IT. Don't wait for two weeks. Either helping you with these requests is not much of a burden, or your supervisor will grow tired of this and do something about it.

Improve this answer
1

Was this a phone call or an email? I find things usually get done faster if you call rather than email. If the communication is time-sensitive you should especially always call instead of email. An email can easily be missed or ignored until after the deadline.

When it comes to technology most people expect instant gratification. Technology is supposed to speed things up and make our lives easier, isn't it? Perhaps it does for the end-user but from an IT perspective it's a much different story. Life is much more complicated for the developer or IT professional who has to set it all up and maintain the systems. It takes time. Technology is never a 100% perfect solution and there's a lot of complicated and time-consuming stuff going on behind the scenes that most end-users aren't aware of.

Improve this answer
1
  • +1 for synchronous communication. Unlike an e-mail, a phone call cannot be ignored until later (possibly forever). Even better, visit the IT team physically (if both you and them don't work remotely).
    – Trang Oul
    Apr 13, 2022 at 6:31
0

my.back had a good answer, and it sounds like this may be the way to go when working through a bureaucracy where some of the solutions which involve some variation of "use your own hardware" just aren't practical. To add to this, I've found the following technique useful when dealing with a department or person who is willing to help, but does things on their own time.

  1. CC your supervisor on every request. This gives your supervisor a chance to give you feedback if you request something that you might not have realized was not necessarily what you need, or should be asking, but also lacking a response from your supervisor, it let's the person doing the thing that needs to be done know that your supervisor is aware of the request. (make sure you follow the advice given by my.back when making this initial request as far as why you need it, and when you need it)
  2. When you don't get a response within a reasonable amount of time, send the request again, but make sure you that you put in the request itself "Second Request", making sure to again cc your supervisor on the request. This lets them know that you still do need this thing to be done, it tends to have an effect of increasing the urgency of the IT professional for your task. Repeat as necessary, making sure to put your request number in the request each time.

In my experience a second request is generally all that is needed when using this technique unless the department is just completely swamped. The tricky part is to figure out what time frame is reasonable to complete such a request, and since organizations differ, it's not one that can be answered here. I would suggest getting to know the folks who are completing your task, so that you can get an idea of what they go through, and so they can see you aren't just asking for ridiculous things and just being demanding all the time, because sometimes it's hard to see the world through other people's eyes without some context.

Improve this answer
0

A simple solution, just always make the requests through your promotor. Maybe explain to them the pattern you have noticed if they ask why you are going through them consistently.

Alternatively maybe they are doing something different, ask them how they get the IT department to resolve the issues quickly.

Improve this answer
0

As a 2022 answer, I suggest using a controlled environment for your software. For example, you might be able to lock down all of the programs you need with Conda (which, does not require admin install rights on many machines). Or, ask IT to install and setup a container program like Singularity or Docker.

Also, consider looking elsewhere. You might be able to use Open Science Grid or similar program depending upon your country and university. Lastly, look for advanced computing centers or similar programs at your university such as Wisconsin's Center for High Throughput Computing or the High Performance Computing Center at Texas Tech.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .