DEMEX Replacement

[scottb613]

Hi Folks,

Just breaking out this discussion to stand on its own for a place to discuss our options.

I looked at the National Map site - it looks like they have national coverage of 1/3 Arc Second or 10 Meter data in GEOTIFF format. This is far better than the 1 Arc Second or 30 Meter data from the Shuttle Mission (SRTM).

https://apps.nationalmap.gov/downloader/#/

So - at some point we are going to need a DEMEX replacement - I have it working on Win10 but others have reported issues with Win10 and Win11.

Just a thought - Blender can import DEM - perhaps there would be a way to script turning the imported DEM into a viable terrain. Well beyond my means.

Regards,
Scott

[pwillard]

Well, is there someone who knows how the MSTS Route Tiles work? I sure don't.

I think Blender could create whole tiles at a time.

(Take a look at how Assetto Corsa tracks are created in Blender... we should have it that easy)

[eric]

If ORTS and TSRE can interpret what Blender produces, great. But that's the rub with most things.

The files used for terrain are compressed binaries. I'll see if maybe there's a way to dump the output TSRE reads in to a human readable format. Maybe someone more math minded can make sense of it.

[scottb613]

Hi Folks,

Alternative to Blender - just using GEOTIFF.

Step 1:

A simple C program (untested) to read the embedded Meta Data (location and positioning data) on a GEOTIFF. Need the GDAL libraries loaded.

#include <stdio.h>
#include <gdal.h>
#include <cpl_conv.h> // for CPLMalloc()

int main(int argc, char *argv[]) {
    if (argc != 2) {
        printf("Usage: %s <GeoTIFF File>\n", argv[0]);
        return 1;
    }

    const char *filename = argv[1];

    // Register GDAL drivers
    GDALAllRegister();

    // Open the GeoTIFF file
    GDALDatasetH dataset = GDALOpen(filename, GA_ReadOnly);
    if (dataset == NULL) {
        printf("Failed to open file: %s\n", filename);
        return 1;
    }

    // Get image dimensions
    int rasterXSize = GDALGetRasterXSize(dataset);
    int rasterYSize = GDALGetRasterYSize(dataset);
    printf("Image Dimensions: %dx%d\n", rasterXSize, rasterYSize);

    // Get GeoTransform
    double geoTransform[6];
    if (GDALGetGeoTransform(dataset, geoTransform) == CE_None) {
        printf("GeoTransform:\n");
        printf("  Origin (Top-Left): (%f, %f)\n", geoTransform[0], geoTransform[3]);
        printf("  Pixel Size: (%f, %f)\n", geoTransform[1], geoTransform[5]);
    } else {
        printf("No GeoTransform available.\n");
    }

    // Get projection
    const char *projectionRef = GDALGetProjectionRef(dataset);
    if (projectionRef != NULL && projectionRef[0] != '\0') {
        printf("Projection: %s\n", projectionRef);
    } else {
        printf("No Projection information available.\n");
    }

    // Read metadata
    char **metadata = GDALGetMetadata(dataset, NULL);
    if (metadata != NULL && *metadata != NULL) {
        printf("Metadata:\n");
        for (int i = 0; metadata[i] != NULL; i++) {
            printf("  %s\n", metadata[i]);
        }
    } else {
        printf("No Metadata available.\n");
    }

    // Close the dataset
    GDALClose(dataset);

    return 0;
}
​

Regards,
Scott

[scottb613]

Hi Folks,

Poking a bit deeper into this - would appreciate some enlightenment if anyone is in the know.

So - for terrain mesh - we're really dealing with the "T" and "RAW" files - correct?

The T file lays out the mesh grid system.

Does the mesh we have in ORTS support a full 10-meter resolution available in the GeoTIFF?

The RAW file is a tile sized chunk of the GeoTIFF that corresponds to the respective grid providing elevation data.

Are RAW files needed once the route is saved?

It seems we should be able to make RAW files from a GeoTIFF using GDAL.

gdal_translate -of ENVI input.tif output.raw
​

We're only talking full tiles here - not the smaller divisions used by the TERRTEX - correct?

It's a shame DEMEX was never made Open Source. Is the author around anywhere?

Thanks.

Regards,
Scott

[pwillard]

I wish I knew more, I'd love to help... I just don't know this stuff.

[scottb613]

I wish I knew more, I'd love to help... I just don't know this stuff.

Hi Pete,

Perhaps I'm diving down this rabbit hole a bit too quick - I think it's time to leave this to people more qualified than I.

I uncompressed a T file and I see it calling RAW files - so they are probably needed.

Regards,
Scott

[eric]

Only the y_raw is needed ----- the e_raw and n_raw files were used by MSTS to build terrain buffers, but that's not used by ORTS.

I've also found out the hard way that the raw files are images...




I've since found this thread from.... 2001. It might help explain the file layout a little better.

https://www.trainsim.com/forums/foru...rting-dem-data

[scottb613]

Only the y_raw is needed ----- the e_raw and n_raw files were used by MSTS to build terrain buffers, but that's not used by ORTS.

I've also found out the hard way that the raw files are images...




I've since found this thread from.... 2001. It might help explain the file layout a little better.

Hi Eric,

Thanks much - I'll take a look. Yeah - after all these years - I'm surprised most of us don't know more about this stuff.


Regards,
Scott

[pwillard]

After all these years, I'm surprised that most of us don’t know more about this subject.

This highlights how much we have lost over time—similar to how master machinists have been replaced by automated machines, leaving only a few people who genuinely understand that technology.

And really, 99% of us were trying to get things to work, while that 1% understood MSTS's inner workings and made tools to improve our lives. Sadly, the "smart" ones also don't seem to stick around... and, as you mentioned... never seem to give up their knowledge (or code). (Probably because it would be over the heads of 99% of the people with access to it)

For example, I am not a programmer, but that doesn't mean I didn't look at GOKU's code. It also means I didn't understand the logic flow... so I quickly got lost.

[scottb613]

Hi Folks,

Whoa - that's some read - that's where John seemed to figure this all out - to create DEMEX. Just perused the topic - it's like having a someone's recipe book. This should be archived somewhere more readily accessible - for posterity.

We probably have tools available today - that they didn't have back then - like the GDAL libraries. Still - this would take a great deal of effort to figure out and is above my paygrade.

Overall steps seem to be:

• Read the GeoTIFF Meta Data to get a valid file description.
• Determine the Tiles needed by the Route.
• Slice the GeoTIFF up into Tile sized bits.
• Convert each bit into RAW using the proper naming convention from the sim.
• Link to the RAW in the T file.

Greatly simplified - of course.

If we had the source to DEMEX we could probably convert it to a 64-bit program which might remove some of the size limitations we encounter.

What's the official status on DEMEX - is John still selling it anywhere? John mentioned not being a cartographer or having a background in same - so - maybe others could figure this out.

I found a separate thread where Sean was discussing 1m DEM and someone posted a procedure on how to use it - overlaying it on top of 10m DEM - but - the US coverage of same is pretty sparse. This indicates that our mesh should be more than capable of utilizing the full resolution of 10m DEM.

Regards,
Scott

[wacampbell]

...,
This indicates that our mesh should be more than capable of utilizing the full resolution of 10m DEM.

Yes, OpenRails terrain resolution is 8 meters.

When using 10 meter DEM, there is some loss in resampling it down to 8 meters ( ie there is a lot of extrapolation being done to interpret values between the points ). But overall, 10 meter DEM is a good fit for OpenRails.

I have tried the 1 meter DEM on some routes, and by the time it gets resampled to 8 meters , it doesn't provide much advantage. Most of the detail is lost.


Wayne

[scottb613]

Yes, OpenRails terrain resolution is 8 meters.

When using 10 meter DEM, there is some loss in resampling it down to 8 meters ( ie there is a lot of extrapolation being done to interpret values between the points ). But overall, 10 meter DEM is a good fit for OpenRails.

I have tried the 1 meter DEM on some routes, and by the time it gets resampled to 8 meters , it doesn't provide much advantage. Most of the detail is lost.


Wayne

Hi Wayne,

Thanks - I've wondered about terrain mesh resolution many times.

8m mesh resolution - with 10m DEM resolution works for me - very cool.


Hi Folks,

The current format of GeoTIFF from USGS is not formatted correctly for direct use in DEMEX - I believe you need to process the files with VTBuilder or QGIS - so - making things easier for newb's to get started - should be a consideration for any replacement or mod. Being able to directly read the current USGS DEM GeoTIFF format would be a huge plus.


Regards,
Scott

[pwillard]

Being able to directly read the current USGS DEM GeoTIFF format would be a huge plus.

Huge agreement from me.

Not sure if this code has any insights...

https://github.com/FrannDzs/Route-Riter

And here

https://github.com/FrannDzs/QGIS

[landnrailroader]

Orignally, DEMEX read the geotiff files as downloaded. Around 7 years ago, USGS changed the format and now if you try to import a raw geotiff, you get some error which I thought would be fatal until just on a whim, I read it into VTbuilder, and then exported it as a geotiff. When you do this the new file is slightly smaller than the original and so I ass/u/me that something is removed. The problem is that at any moment, and best as I could tell, no notification, USGS changes the format. So what then. Maybe VTBuilder would fix it, maybe not. Also, VTbuilder will not read the 1/3 arc-sec files, only the 1 arc-sec, so you cannot get really good resolution. In fact you might not get any better than .hgt files, but so far I have used only the geotiff files.

definition: Every Southern Rwy officer, no matter how low on the totem pole, was required to keep a little sign on his desk. ASS/U/ME This was to remind you that if you assumed anything you were making an ass out of yourself and the other individual involved. I kept it on my desk at CSX. There was another: YCSFSOYA Every officer was to represent the company as best as they could especially around shippers, so YCSFSOYA = You can't sell freight sitting on your ass.

Jerry Sullivan