19

I am a physicist currently doing a master's degree and I have a background in programming with a knowledge of computer science — not on the level of someone who studied computer science but I would still say a bit above average.

As I program a lot for physics and astronomy applications, I have become increasingly interested in how to further expand software and scientific software to be more usable for natural sciences and physics in particular. Now I would like to know whether there is a specific sub-field that addresses this.

Examples of what I am interested in include:

  • how to include units in a standardized way in file formats specifically made for storing scientific data (e.g., HDF5)
  • whether it would make sense to create a programming language with the specific intent of making scientific calculations easier (including units and measurement errors standardly)
  • how to create better plotting solutions that can be edited after they have been created. (Similar to Matlab .fig files but not proprietary)

It would be super helpful if anyone knows something here.

17
  • 9
    This feels more like an entrepreneurial pursuit than an academic field to me. Mar 10, 2023 at 16:59
  • 8
    In what way would you consider this to be entrepreneurial? My interest lies in the applications to science with free and open standards aswell as software. My intent is not to make money but in advancing science by providing newer and better applications and standards. Mar 10, 2023 at 17:08
  • 2
    You might find the Programming Language Implementation and Design Stack Exchange site useful. (It doesn't exist yet, but it probably will in a few months.)
    – wizzwizz4
    Mar 11, 2023 at 2:45
  • 12
    I'm not sure about standardizing, but the type of folks who work on research software are, appropriately, called "Research Software Engineers". See here: us-rse.org Mar 11, 2023 at 6:18
  • 3
    There is a lot of niche scientific software out there that's some PhD student's pet project from several years ago. Generally they are held together with just enough sticky tape and chewing gum for said student to get enough data to pass their doctorate, so expect some instability and incomplete functionality. Thankfully with the rise of open source research group software repositories on sites like GitHub and GitLab, a lot more of this software can be improved and made more robust by future researchers in the field. Mar 12, 2023 at 15:48

11 Answers 11

33

I think what you are looking for is "Research Software Engineering" (RSE). Currently this is more of a job title than a discipline, but RSEs are starting to organise more and more like a discipline: they have their own conferences, learned societies and journals, etc.

Its a concept that started in the UK within the last decade, but is spreading internationally now. See https://society-rse.org/

For institutions that buy into the model properly, there is often a separate RSE team or lab. In my university this sits within the CS department, but most of those in the RSE team started as domain experts in something else, like Physics, Chemistry or Biology and had been involved in code based research in those disciplines.

15

In my experience, attempts to standardize scientific software fall into this problem:

https://xkcd.com/927/

When you're doing research, it's really difficult to have a set of standards that actually apply to everyone, because everyone is asking a slightly different set of questions, and those questions may not be compatible with previous solutions in how to organize a problem or organize data.

The easiest places to address are where specific equipment used in a field dictates a particular format. If everyone is using the same machine that produces output in a ".sci file", there will be a need for software that operates on ".sci files". The same happens if there's a particular open source software that everyone uses. I doubt, though, that you'll find anything standard to "physics" or even "astronomy"; rather, you'll find standard usage within a specific subfield of physics or astronomy, where people are asking and answering related questions and sharing data amongst others working on the same problems.

Besides that, there's always going to be a trade-off between ease of use and flexibility of use.

16
  • 3
    Ah i love xkcd. Thanks for reminding me of this. Sometimes I realy like to overstandardize, but i think there were alot of attempts of creating standards that were actualy sucessfull and decluttered everything, take USB for example. Mar 10, 2023 at 20:08
  • 1
    @BryanKrause. I miss the days of stacking 9 to 25s and 25 to 9s, with various male/female arrangements and some both wothcand without null modems, maybe withca security dongle or three thrown into the mix, and diddling around until you made ccx a connection. Mar 10, 2023 at 22:02
  • 5
    Speaking of USB, despite the physical USB-C connector being standardized, we’re now seeing a plethora of different standards and capabilities, so even if you can physically connect something you don’t necessarily know that what you want to do will be supported …. various things like data transfer rate and maximum power draw, among others Mar 11, 2023 at 12:30
  • 2
    @ScottSeidman u wot m8? Mar 11, 2023 at 19:40
  • 3
    @ScottSeidman It was a joke. Without field specific knowledge your comment is completely impenetrable. Mar 11, 2023 at 21:09
8

You are, in fact, asking about applications of certain fields, including CS, to a certain class of problems.

It is natural, in applied fields, to bring a variety of tools to bear depending on the needs of the specific problem. Those tools may have (probably have) a theoretical basis, but the problem at hand may be very messy.

So, the fact that you may need programming in some general or specialized language, statistics, graphics, human factors, etc. etc. etc. isn't at all surprising. Teams of people with different backgrounds and skills is often a productive way to proceed. CERN, for example, has lots of specialists in lots of specialities and their expertise is brought to bear, or not, according to the needs of the day.

If a class of problems (applications) is both important and recurrent, then specialized tools might arise in solving one that might be used in other, similar, problems, but the world as we know it is still messy and requires some ingenuity to deal with.

6
  • So basically the Discipline i would be thinking about is Computer Science or maybe a sub-field thereof? Mar 10, 2023 at 20:24
  • More like "applied CS" which depends on the normal tools of CS to solve real world problems. It isn't really anything special beyond that.
    – Buffy
    Mar 10, 2023 at 20:25
  • 1
    I have not looked into the correct direction here, as i was just looking at Physics and subfields like Comp Physikcs. My main goal was just to find a way to describe what i am interested in and to find if this was an existing field. Thanks for your answer. Mar 10, 2023 at 20:28
  • 3
    @TheMastermage More like a sub-sub-subfield of applied CS. There's been plenty of attempts to e.g. incorporate robust unit/dimension handling into programming frameworks (Boost comes to mind) but this is very much a practical concern rather than an academic discipline. Oh, and I'd suggest that "inventing a language" is something completely outside your scope, at least until you have completed a comprehensive review of the many thousand that exist already... with perhaps the most recent notable attempt being Rust. Mar 11, 2023 at 10:54
  • 1
    @TheMastermage well, it was your OP that mentioned creating a language :-) I think possibly I might pick up on that as an answer, since I think there is not enough space in this margin :-) Mar 11, 2023 at 16:41
7

Upgrading from my comment, a don't think you'll find a scientific or academic discipline that aligns perfectly with your goals.

You seem a little bothered by my description of your description as more "entrepreneurial". Don't be. It's not meant as an insult. There's nothing wrong with an entrepreneurial pursuit. Plenty of great things happen because of clever entrepreneurial people. You may not be planning on making money, but I suggest even the person with the purest motivations in the world needs to eat and keep the lights on, and if one can't find a way to do that while making the world a better place, then one needs to keep thinking.

The reason why I call what you want to do as entrepreneurial is that you seem interested in provided a service or tool to people in scientific or academic pursuits.

That said, there are some academics (I think) that do some work in the visual representation of data, like Tufte (https://www.edwardtufte.com/tufte/books_vdqi; https://en.wikipedia.org/wiki/Edward_Tufte) Note though, that I don't believe Tufte was into making his tools available to academics. There are, however, many companies that have incorporated many of Tufte's ideas and recommendations into their products (SigmaPlot, Prism, ...) They do this to make their business models work.

Now, think about how tools get to investigators. I believe the most frequent models is that somebody does work, becomes expert in it, and then build the perfect tool for them and their group. They find a way to make it available (think ImageJ, or just about every R module). If it's a good enough tool, with enough of a user base, maybe a company will pick it up, and package and sell it. The people that develop the tools from the ground up usually aren't in the business of tweaking the tools. In fact, no academician would want to be saddled with packaging, distributing, and (perhaps hardest), supporting the application. That requires a business, with a staff, a budget, the logistics, ... -- repeat business.

I'm not even beginning to imply that you can't have an absolutely marvelous and rewarding career doing what you describe. I just think you'll have the highest likelihood of making it work as a business model. I'd encourage you, if you want to pursue this area, to actually build a business model, figure out who your customers would be, the size of the staff you would need, how many instances you would need to sell, what you would need to charge for them, how much seed money (diluting and nondiluting) you would need to set up this business, where are you most likely to find this money, how long before your company would have a positive cash flow, etc.

Alternatively, you can find a company already in the area, or close to the area, and see if you can find employment in there. Convince that company that with the right sort of hire, they can maybe expand their customer base with little risk if they expand their product line, and then do your magic on their dime.

14
  • 2
    So far my plan has not been on creating my own company while I highly respect those who do become self employed it is not the way for me. I will take your comment to heart tho especially your last paragraph. :D Mar 10, 2023 at 20:17
  • 2
    The Mastermage: Regarding "entrepreneurial" -- Given the extremely high visibility of Stephen Wolfram's Mathematica in physics, I would imagine you're aware that his original motivation sounds very much like yours: "I started out—in my teens—as a physicist. And doing theoretical physics requires doing lots of mathematical calculation. And I figured: these calculations are kind of mechanical. (continued) Mar 11, 2023 at 9:12
  • 3
    @TheMastermage I agree this sounds like a business venture to me, because I think you dramatically underestimate how large a team of programmers would be required to create a whole, new, useful, supported and adopted scientific programming language. If you really think there are specific things missing, create an R package. If you really think standardisation is missing, make sure to republish your package for R, Python, Julia, SAS, MATLAB, STATA, SPSS, Maple, Mathematica, etc. Then at least most people can use your new 'standard'.
    – niemiro
    Mar 11, 2023 at 9:50
  • 1
    @niemiro or publish it in C++ with a C interface, because most (all?) of these languages can link to C with relatively little effort. The other way around, try using a piece of code written in, say, Matlab from another language, and you'll be in for a much more painful experience. Mar 11, 2023 at 11:34
  • 1
    @TheMastermage not quite, but you can compile them to object files that can be linked to an executable as if (-kinda) they were written in C. Simple example in Haskell: wiki.haskell.org/Calling_Haskell_from_C (This one uses GHC for compiling also the C main module; that makes it easier but is not strictly necessary.) Mar 11, 2023 at 16:48
7

Is there a field which specifically addresses how to develop new software specificaly for natural sciences?

... further expand software and scientific software to be better usable for natural sciences and physics in particular

What I understand you're alluding to is #reproducibility, alongside #reliability and #validity

As a role, these are scattered across: computational biologist, computational physicist, computational scientist, lab scientist, research scientist, research engineer ... even software scientist.
Of late, they are 'morphing/converging' into research software engineer/scientist with a path to research technical manager.
Consequently, this also brings in 'new field/discipline' loosely referred to as Research Software Engineering.
[For me though, it might be inconsequential if it's research software engineer or research software scientist].

RSE as a discipline is getting well entrenched in the UK, in the States and the rest of the world catching up fast. Invariably, it's gaining ground globally.

In that regard, RSE are known as people with a specific set of skills that combine the research knowledge and software engineering. A fear I have is RSE becoming 'just' another SE in a 'vertical industry' (which just happened to be Research or R&D) and hence loosing out the research software scientist or the science/research of the research software.

A paper on RSE - Woolston, C. (2022). Why science needs more research software engineers. Nature.

Alliance/funding org for RSE - ReSA

Bodies for RSE:
Aligning together internationally International Council of RSE Associations / Research Software Engineers International

In the UK for instance, apart from university research centres, RSE abound in places like STFC, DiRAC ...

"Software is the ubiquitous instrument of science. - Carole Goble, Professor of Computer Science, University of Manchester, UK"

6

There is the Scientific Python community:

The Scientific Python ecosystem is a loose federation of community developed and owned Python projects widely used in scientific research, technical computing, and data science. This website is part of the Scientific Python project, which aims to better coordinate the ecosystem and grow the community.

At least for the examples you provide, there are existing and actively developed software solutions. A very incomplete list:

  • xarray "[...] makes working with labelled multi-dimensional arrays in Python simple, efficient, and fun!" It is used widely in climate science, and comes with Dask support out of the box.
  • Pint and Astropy provide physical units. Integration of Pint with Xarray is actively developed further.
  • NetCDF and NeXus specify how to store units in files. Both are formats built on top of HDF5, as per your question.
  • Scipp supports multi-dimensional data arrays with labeled dimensions in a manner similar to Xarray. It is a Python library enabling a modern and intuitive way of working with scientific data in Jupyter notebooks. It supports units out of the box.

Disclaimer: I am the lead developer of Scipp.

6
  • 1
    Scipp looks good. Well done! Mar 12, 2023 at 17:52
  • 1
    as a heavy user of the whole SciPy stack, I am very grateful to them and they are doing an incredible work to the scientific comunity. May i specifically ask what Scip does differet than xarray? They seem very similar? Mar 12, 2023 at 21:21
  • 2
    @TheMastermage Scipp and Xarray are indeed similar. Scipp's direct support for units provides a considerably better user experience. It is fast and multi-threaded out-of-the-box, whereas Xarray users have to rely on dask, which is more of an expert tool but allows for scaling beyond a single machine. Finally, Scipp provides algorithms and data structures for non-destructive binning and histogramming, for irregular data such as event data. Xarray is more mature and feature rich in other areas, but also more complex, in part due to its Pandas-legacy.
    – Simon
    Mar 13, 2023 at 5:31
  • See also What is Scipp and Comparison with other Software.
    – Simon
    Mar 13, 2023 at 5:32
  • 1
    @Simon, this sounds realy cool thanks for the explanation. I will probably be using Scipp in the future. Mar 13, 2023 at 15:34
6

The closest thing I can think of as a discipline or field would be computational biology, but it's still not really getting at what you're thinking about. I actually spend a lot of time thinking about the same things as you: making better tools for scientists. And Scott Seidman's answer really gets at what that looks like in many cases. Scientists have a project idea, and a tool comes of it as a side-effect, and sometimes they try to share it with others. I've used a lot of these tools, and often, they have a lot of room for improvement. In my field (ecology), there are definitely job opportunities for developing and/or improving research software at all levels (non-profits, business entities, academic research groups, and local/state/federal government agencies). This is what I do right now; I volunteered to develop and publish a software package some years ago as a side project during my PhD, and now I'm employed part-time by an academic group to continue developing and improving it, as well as have a contract with a government agency to add some novel features to it. The funding from the academic group came in part from a research grant that specifically supports developing research software, and now we are working on getting an NSF grant that would support me full-time for a few years to keep working on the same software. There are tons of research grants available for this type of work; it's just a matter of finding the right research group to work with.

In the meantime, I have some software side projects that I work on in my free time that are aimed at addressing other issues I see that aren't really specific to a discipline. Some are small and aren't really big enough to warrant funding; they are more just fun toy projects that will help to build out my portfolio, and maybe a few people will find them useful. But I have a big one that's kind of generic and might be a little harder to find a research grant and group to focus on (if I wanted to go that route). Instead, I'm hoping to turn it into an entrepreneurial opportunity down the line when it is more stable.

I've also considered applying for jobs at businesses that do this type of work. A good example is Posit (formerly RStudio). They don't just develop an IDE, they develop a ton of R packages and other tools with a focus on improving scientific computing. Then there are other businesses that develop software with more specific scientific applications in mind. The only reason I haven't gone this route (yet) is more related to why I got into programming in the first place (long before I did any kind of scientific computing) and my desire to develop my own ideas.

But I do have one piece of advice: be careful about thinking you have "better" solutions (like file formats, plotting, etc). First, we all live in a bubble and often don't see many things that already exist, so there's a good chance someone has already had the same idea and implemented it somewhere, and you just haven't found it yet. Second, it's really hard to convince people to try new things; it can take a lot of time and effort for people to learn new tools and modify their workflow around them. Even if the long-term payoffs are worth it, budget limits, other deadlines, and just plain other things they'd rather do get in the way of your new software gaining traction.

I say this as someone that has had tons of ideas related to programming language design (I have yet to encounter a scripting language that I like using for scientific computing), file formats, data management, plotting tools, etc. I toy around with these ideas with the hope that maybe some of them are novel (or at least a better implementation) and people will use them someday, but I basically assume that they don't have value until they exist and are widely used. In the meantime, as I said, I depend on working with research groups to develop more specific tools to keep me funded.

2
  • Thank you for your deep industry/science insight here, gives me quite an interesting perspective. I will take your advise to heart, I dont wish nor think i am the one who has (as we in germany call it) eaten wisdom with a spoon, so all of these things i mentioned are more like ideas, that i think could have merit. But I am open to discussion and in fact that is the reason why asked for a field here, to find the people with whomst I can discuss this. Mar 13, 2023 at 15:59
  • I also dont have any illusions onto the fact that people, me included, are very inert when it comes to changing to new software. The reason I stayed with python and matplotlib the past couple years even tho I dislike them in certain areas. I also sometimes would rather like to expand on existing things with my ideas if possible. So If I will write some side projects then in the knowledge that it might never gain traction but atleast it will maybe satisfy my needs. And maybe someone else will find them usefull, which I realize is exactly how projects go as you had mentioned in the begining. Mar 13, 2023 at 16:05
6

TL;DR:

The field involved in development of software for the purpose of research is called Research Software Engineering (RSE). In the USA one resource for more information is https://us-rse.org/.

Details

I can't speak to standardizing research software. I'd caution that any attempt to standardize research software is fraught. After all, if we knew exactly what we were doing, it wouldn't be called research.

That said, there are definitely standards to developing research software, at multiple levels. For context, I have worked as an RSE and currently work as a research computing facilitator. I have built or facilitated a wide variety of research software solutions. The list that follows comes from my personal experience.

  1. Numerical computing. The field of mathematics involved with understanding the behavior of approximate mathematical calculations in the context of finite representations of real numbers, i.e. floating point numbers. Practically, this informs us how to develop mathematical algorithms that will converge to an error tolerance as quickly as possible, rather than compounding floating point errors.

  2. Software engineering best practices. Things like version control, unit and integration testing, issue tracking, developer and user documentation, minimize complexity, maximize readability, appropriate abstraction, assert pre/postconditions and invariants, code reviews, continuing education, etc.

  3. Verifiability. Does the code produce outputs that are expected? This is distinct from unit and integration testing. The goal of testing is to check parts of the code do what they are intended to. The goal of verification is to check that the software output matches what the science says the output should be. This can be deceptively hard!

    Each scientific domain may have a different approach to verification. Some may have multiple approaches, which can be a research subfield in its own right. Some may have not developed an effective means of verification.

    Some may use rigid comparison of results to know values. Some may use stochastic comparisons, i.e., checking a distribution of outputs matches a known distribution. Sometimes a sensitivity study is suitable, i.e., varying inputs in a systematic way to observe the behavior of outputs. Very much like a Design of Experiments. Some may have to compare to proxy models.

  4. Reproducibility. How easy is it for someone else to set up and reproduce your results using your code? There are myriad tools for this at different levels. RNG seeding, Anaconda environments, containers (Docker and Singularity), workflow managers (Snakemake, Nextflow, Pegasus, etc.), version control.

All of these are things Research Software Engineers should be aware of. Many of them will touch all of these at some point in their career, if for no other reason than they solve a lot of problems and make software engineering and deployment work easier.

If this sounds like the type of thing you are after, then you are absolutely looking for Research Software Engineering.

Physical Units Management

There may be various packages out there that manage physical units, and there may be some proprietary libraries used by groups in mission critical domains (Aerospace, Medical) to avoid costly (or deadly) errors.

For high-throughput applications, automatic unit management at runtime may be an unnecessary burden. For statically-typed languages, the types could be used to check unit errors at compile time, which could be fantastically helpful for catching errors early. However, you'd need to design a lot of classes (or use something like template metaprogramming) to allow changing types through multiplication. This is definitely possible in, for example, C++, but the up-front design and maintenance costs may be substantial, and it adds burden to developers.

Oftentimes in practice it's easier to use a modified Hungarian notation like the following block.

width_mm = 3
height_mm = 2
area_mm2 = width_mm * height_mm

mm_to_m = 10^-3
area_m2 = area_mm2 * mm_to_m^2

length_cm = 100
volume_mm3 = area_mm2 * length_cm

It should be immediately obvious from the variable names which lines are correct and which have errors. I'll leave it to you to figure out how to denote negative powers of units.

Bonus Information!

If there is any chance your research software will need to scale up, be sure to consult with your local Research Computing (RC) or High Performance Computing (HPC) staff before and during development. Most academic institutions have some amount of RC/HPC staff. They can help you design your work to ensure less friction when scaling up.

RSEs often work closely with RC/HPC folks for exactly this reason.

It may be helpful to read this recent blog post on what research software is: https://upstream.force11.org/defining-the-roles-of-research-software/. It has a categorization of research software into seven different types. The blog post goes into greater detail on each.

  1. Component of research instruments
  2. A complete research instrument
  3. Data analysis pipeline
  4. Data presentation or visualization tool
  5. Integrates components into a whole (glue code, workflow management)
  6. Is infrastructure, a platform, a framework, a library, a language, etc.
  7. Facilitates research-flavored collaboration

Summary

It is clear that there are many different types of research software, fulfilling many different roles and functions. This huge variety makes it hard to come up with a good classification that captures all aspects and does justice to all the hard work done by the developers of the software. Nevertheless, we hope that we have succeeded in proving a bit more insight into the value of research software, the importance of sustaining said software, and recognizing the people involved in developing the software.

3
  • Thank you for your input on this especially the nice list up of the different things to consider here. I do not plan on standardizing software as a whole but I am interested in Standards that facilitate an easier exchange of scientific data e.g. unit support. Mar 13, 2023 at 17:02
  • 1
    Thanks for the kind words. I'm not familiar enough with scientific programming of units to speak to it in great detail. I'll add a section with my thoughts at a high level. Mar 13, 2023 at 19:08
  • @TheMastermage You may find this blog post illuminating as well about what Research Software is, at a high level. I'll add it in the bonus section to keep it around long term: upstream.force11.org/defining-the-roles-of-research-software Mar 14, 2023 at 14:17
4

Upgrading one of my comments into an answer, in two parts:

First, several other answers mention visualization as a sub-sub-sub-field of computer science. That's correct: Data visualization is a subset of computer graphics, which is a subset of computer science in general. However, even sub-sub-sub-fields of computer science can (and do) support vibrant research communities. One can obtain a PhD with a focus exclusively on data visualization, and there are several respectable conferences in the field every year.

That said, I don't know that Data Visualization gets you fully where you want to be. It's certainly an area to investigate, though. And DV tends to be interdisciplinary (data does not exist in a vacuum, and it helps to understand the challenges of another research field before you go charging in trying to solve their visualization problems for them) which may be a definite plus for you.

Second, the notion of trying to include units (I think you mean physical units) into software brings to mind the last well-publicized push in that direction, at least the last one I recall: The Fortress Programming Language. It was a major effort which, alas, did not pan out. It was shut down about ten years ago. Surely these issues are faced all across many industries, as well-- mechanical simulators, electromagnetic simulators, astronomical software, particle physics software, etc. Unfortunately, I do not know anything at all about how they handle it.

But there are areas of computer science dedicated to programming language design and compiler design. And that's just a sub-field, not a sub-sub-sub-field. These will also be related to "scientific computing" at large.

2
  • Thanks for the expansion on this, I havent heard about Fortress so far. As with all new programming languages it always difficult to get people to adopt it especially when it is not used much yet which sadly leads to the decline of some interesting ideas. Mar 13, 2023 at 17:08
  • @TheMastermage well, it's even harder to get people to adopt a language which could not be made to work....
    – Anonymous
    Mar 14, 2023 at 3:25
3

Apropos "creating a language", there are two aspects here:

  • Representational languages
  • Computational languages

The dominant trend these days is for newly coined representational languages to be based on XML, often zipped, the practicalities of which are thoroughly "applied CS". The details of this- the schema etc.- are highly application-specific, and are likely to be very different for e.g. 3D particle tracks captured at CERN vs statistical analysis of broadband noise on a communication channel.

There are of course exceptions, and I'd highlight the work that Lenat has done representing ontologies etc.: I'd not like to attempt to comment on whether that could have been done using e.g. PROLOG.

As far as computation goes, there is a regrettable tendency these days to assume that Python is adequate for all possible cases (obligatory xkcd: https://xkcd.com/413/), even if when one scratches the surface one finds that all it's doing is pulling in some decades-old FORTRAN.

However there are a couple of cases which might possibly justify either a custom language or compiled-in extensions to an existing one:

  • Where it is desirable to specify attributes such as units at the language level rather than using an existing type system or as strings to be parsed at runtime.

  • Where practitioners are so used to their custom representation that they are extremely resistant to any language which doesn't support it directly.

As examples here I would offer the PostGIS extensions to the PostgreSQL database, plus of course the continued use of PROLOG in certain fields of knowledge manipulation. However my opinion is that both of these benefit from being able to be embedded in or intermingled with a conventional procedural high-level language such as C++.

So I think that might possibly suggest some areas for research, but the fundamental issues are the structure of the data being manipulated and the expectations of existing workers.

2
  • 1
    "As far as computation goes, there is a regrettable tendency these days to assume that Python is adequate for all possible cases". Sure, Python is not always the best choice. But it seems to have achieved a critical mass : many people in general, and scientists in particular, are using it. Which means that if you google "python + something_you_want_to_calculate_or_display", you'll find a nice, recent library allowing you to do it. And with two or three imports, you can achieve awesome results, which you would have to write from scratch in another language otherwise. Mar 12, 2023 at 17:56
  • Very interesting look at this, and I do agree on the Python part. Mar 13, 2023 at 17:24
1

Do not invent a new language

The idea that a different language will save you is a popular misconception from CS. Sure, you can add features to a language. But it doesn't buy you anything over adding a library to do those things.

And by the very definition of creating a new language yourself, you've limited the number of people who will ever use it to one. Yourself. Software engineers have had 60 years to write languages, and you're highly unlikely to find a new niche and get critical mass of new users. The surest way to guarantee no-one else will ever use your work is to invent a new language for it.

What's wrong with making these features a library?

You're describing having units and tolerances/uncertainties attached to a value. This is literally a beginner's C++ exercise for writing your first class. If you're not aware of OO (object orientation) principles, I strongly recommend reading up on this. It's key to how pretty much all modern languages work.

What's wrong with existing languages?

Industry overwhelmingly uses C++ or C# for historical reasons, and because they run fast. Rust is a new entrant in the same area, and it may pick up enough to get popular since it was adopted by the Linux community, but it's still well behind. For server-side work, you'll meet JavaScript. In the maths/science community there's a strong user base for MATLAB, again for historical reasons (and because it does what it does very well). You may also meet R and Julia in academic maths/science stuff, or Fortran for older stuff, although they're less common. And Python is popular everywhere for writing programs quickly, but it runs slowly (because it's an interpreted language).

When you know enough about those languages to know features that are missing from all those languages,and you know they can't be added as libraries, and the language maintainers reject your request for those features, then you may have a case for writing your own language. I'm going to say that's pretty damn unlikely though.

3
  • My main idea with this is, there of course are unit libraries for most of any Language available but this adds extra dependencies, and why add extra dependencies when you could have this shipped by default. I am open to discuss the merits of this. I dont quite agree with your reasoning, I think new languages can be developed even if they dont reinvent the wheel. Just because there is 60 Years of development doesnt mean there can't be new programming languages. Look at Carbon for example, everything could also be written in C++ but Carbon builds up on the positives and adds modern features. Mar 13, 2023 at 17:32
  • "The idea that a different language will save you is a popular misconception from CS..." While this is a pernicious trap, an even worse one is investing time and effort into an improved compiler for an existing language because some feature of the syntax makes it difficult to parse. I submit that if a computational language is difficult to parse it is better to fix the language than to try to write an improved compiler, and by extension would suggest that if a representational language is difficult to process one should look for flaws in the schema describing the data layout, Mar 13, 2023 at 17:51
  • @TheMastermage What's your problem with dependencies? Unless you're writing embedded software running on raw metal with no OS or anything else, you've got dependencies up to your eyeballs from the moment you write Hello World. Why could one more dependency possibly matter? Especially since it's something you've written yourself. Or hey, you've got the source code, so you don't even need to make it into a library if you don't want to. You're trying to solve a non-problem.
    – Graham
    Mar 14, 2023 at 1:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .