SCO LIDEX goes GLOBAL

[scottb613]

SCO LIDEX goes GLOBAL
SCO LIDEX Terrain Builder v1.200

Hi Folks,

Well - a new update.

When I first posted about SCO LIDEX, one of the first questions was whether it could build terrain outside the United States. At the time, the honest answer was no. LIDEX depended on USGS elevation data, so our friends' building routes in Europe, the UK, Australia, and most of the rest of the world were left standing at the station.

That answer has changed.

SCO LIDEX v1.200 now provides worldwide terrain-building coverage.

It still uses the best available high-resolution USGS data for United States routes, but it can now fall back automatically to the public Copernicus GLO-30 global elevation dataset. The same release also builds Distant Mountains and can create detailed OpenStreetMap map tiles without making the user manually download and install map images - one by one.

In plain English - point LIDEX at the route, select what you want, run Scan, and let the program sort out which data sources can do the job.


HIGH-LEVEL FEATURES
================================================== ==========

• Worldwide terrain coverage using the key-free Copernicus DEM GLO-30 Public dataset.
• High-resolution U.S. terrain remains intact using the established USGS source sequence.
• Automatic source fallback uses the best available elevation source and fills unresolved terrain from the next viable source.
• Global Distant Mountains use Copernicus GLO-30 exclusively, resampled onto the existing 128-meter low-terrain grid.
• OpenStreetMap map tiles are created automatically from anonymous Geofabrik regional extracts.
• No manual map downloads and no separate map-image installation step.
• No API keys for Copernicus, Geofabrik, or USGS public data.
• Read-only Scan checks every selected data source before LIDEX writes route terrain.
• Clear source reporting shows PASSED or FAILED for each source and explains the difference between a service outage and simply having no coverage for the route.
• Independent production stages allow viable terrain, Distant Mountain, or map work to continue when another source is unavailable.
• Append and Overwrite modes remain available for new terrain and deliberate route rebuilds.
• Resumable OSM downloads continue an interrupted regional download instead of throwing away the work already completed.
• Selective cache cleanup lets the user choose individual OSM downloads to remove when exiting LIDEX.

THE ELEVATION SOURCES
================================================== ==========

LIDEX checks the elevation sources in this order:

1. USGS 3DEP 1-meter DEM - the highest-resolution source used by LIDEX for covered United States routes.
2. USGS Original Product Resolution - shown as 5m~ - used where applicable when 1-meter coverage does not resolve every terrain post.
3. USGS/NED 1/3 arc-second - shown as 10m - the established lower-resolution U.S. fallback for normal terrain.
4. Copernicus DEM GLO-30 Public - shown as 30m (global) - the key-free worldwide fallback supplied through AWS Open Data.

Normal terrain follows that source order. Distant Mountains go directly to Copernicus GLO-30 because their stored grid is already approximately 128 meters per post; DM-only work does not query USGS.

Note: 30-meter global data is not magically the same thing as 1-meter LIDAR. Copernicus GLO-30 is a lower-resolution digital surface model, and its elevations may include vegetation, buildings, and other infrastructure. It gives route builders practical worldwide coverage - it does not pretend every country suddenly has USGS-quality 1-meter terrain.

LIDEX identifies global processing clearly in the interface and log. On a U.S. route, it will still try the finer USGS sources first and use Copernicus only where the selected route needs it.


OPENSTREETMAP MAP TILES - NO MANUAL DOWNLOADS
================================================== ==========

Select Create Map Tiles in SCO LIDEX and use the normal Scan and Run workflow. LIDEX finds the required Geofabrik regional OpenStreetMap extract, downloads it, and creates a 4096 x 4096 map image for each selected normal terrain tile.

There is no separate map website to visit, no pile of images to name, and no manual map installation.

After opening the route, use the MAP TILE SHOW/HIDE button to display or hide the generated map tiles.

That's it.


WHERE LIDEX STORES THE OSM DATA
================================================== ==========

This is new for v1.200.

The large regional OpenStreetMap extract is stored with the route:

<Route>\osm_data\geofabrik\<region-id>.osm.pbf

LIDEX also writes the route's OSM cache description here:

<Route>\osm_data\osm-cache.json

The small shared Geofabrik regional index and the route-cache registry remain under:

%LocalAppData%\SCOLIDEX

In other words - large PBF downloads stay in route folders where they can be seen and managed, rather than quietly filling a hidden application-data folder. LIDEX checks the selected route first, then sibling and registered route caches. If any existing PBF covers the selection, LIDEX uses it directly from that location without copying it. Only when no covering cache exists does LIDEX download a fresh PBF into the selected route. Purge an existing PBF before a run when a fresh download is wanted.

When exiting LIDEX, the cache screen lists regional PBF files and incomplete downloads individually. Every purge box is clear by default. The small shared index is not included in the purge, and LIDEX never deletes the generated map images from the route's terrain_maps folder.


BASIC WORKFLOW
================================================== ==========

1. Back up the route. Terrain work is still terrain work, and a backup remains cheap insurance.
2. Select the route folder in SCO LIDEX.
3. Choose normal terrain, Distant Mountains, and/or Create Map Tiles.
4. Run Scan and review the status of every data source.
5. Run the production stages that Scan has enabled.
6. Open the route and use MAP TILE SHOW/HIDE to view the generated map tiles.

LIDEX can work from existing route tiles or build coverage from a tile list, marker file, KML file, or track database. Scan remains read-only. Run writes only the production stages you selected and that Scan determined had a viable data source.


REQUIREMENTS AND NOTES
================================================== ==========

• 64-bit Windows 10 or newer.
• Internet access while requesting elevation or OpenStreetMap source data.
• Always keep a route backup before using Run, Overwrite, Clean Tile Wipe, or Commit/Post Processing.
• Large routes and regional OSM extracts can require substantial processing time and disk space.
• SCO LIDEX is released under GNU GPL v3 or later.

DOWNLOAD SCO LIDEX v1.200
================================================== ==========

Look under Releases and download SCOLIDEX-v1.200-win-x64.zip:

https://github.com/scottb613/SCO_LIDEX

Thanks to everyone who tested the earlier U.S.-only releases.

Regards,
Scott

[scottb613]

Hi Folks,

NEW RELEASE: v1.300 - see release page for changes.

https://github.com/scottb613/SCO_LIDEX

Regards,
Scott

[scottb613]

Hi Folks,

NEW RELEASE: v1.400 - to easily accommodate smaller monitors. We've gone dark...

https://github.com/scottb613/SCO_LIDEX

Capture3.jpg

Regards,
Scott

[scottb613]

Hi Folks,

Damn - found a bug - it's a complicated program with many moving parts - sorry.

Just download and replace.


V1.401 HOTFIX

https://github.com/scottb613/SCO_LIDEX

Regards,
Scott

[Tonya37]

Scott, I just had a go at V1.401. I started a new route (actually a route that I've worked on before), laid a bit of track, then without reading any instructions I dived straight in. Well! It all worked perfectly, loaded the terrain and the maps. Couldn't fault it. The 30m terrain was a *lot* better than the 50m terrain I had previously used and appeared to be more accurate. It is in an area where I once lived, and looking from where the front garden of my grandparents' house is it all looked very recognisable. Sadly no Lidar for us European peasants so we'll still have to build our own embankments and cuttings, but it looks like no major landscaping will be required as before

I have a couple of queries. I use Library of Scotland Maps dating to pre war rather than the modern OSM. What I do is to copy the maps on to those created by TSRE5. Is there any way to do the same with Lidex? Is it possible to use TSRE5 to load the modified maps as before? (I haven't actually tried any experiments as yet so perhaps I'm jumping the gun.) Perhaps I should avoid loading the maps?

Second question - terrascaping is error prone (or at least takes a lot of trial and error. Obviously one can make a backup of a tile before terrascaping and if one makes a mess use it to recover. I wonder if there is a way to generate dem terrain directly to a single tile?

I haven't tried a 4m mesh just yet but I'm going to try that out next.

This program, without exaggeration, is a game changer. It makes things so much simpler, and I can't express my appreciation enough, I'm sure many will feel the same way. Making it work worldwide means that it is going to get a lot of use. I kept up with your posts while you were developing Lidex and I was wishing it would work for the rest of us too...

Regards

Tony
.

[scottb613]

Scott, I just had a go at V1.401. I started a new route (actually a route that I've worked on before), laid a bit of track, then without reading any instructions I dived straight in. Well! It all worked perfectly, loaded the terrain and the maps. Couldn't fault it. The 30m terrain was a *lot* better than the 50m terrain I had previously used and appeared to be more accurate. It is in an area where I once lived, and looking from where the front garden of my grandparents' house is it all looked very recognisable. Sadly no Lidar for us European peasants so we'll still have to build our own embankments and cuttings, but it looks like no major landscaping will be required as before

I have a couple of queries. I use Library of Scotland Maps dating to pre war rather than the modern OSM. What I do is to copy the maps on to those created by TSRE5. Is there any way to do the same with Lidex? Is it possible to use TSRE5 to load the modified maps as before? (I haven't actually tried any experiments as yet so perhaps I'm jumping the gun.) Perhaps I should avoid loading the maps?

Second question - terrascaping is error prone (or at least takes a lot of trial and error. Obviously one can make a backup of a tile before terrascaping and if one makes a mess use it to recover. I wonder if there is a way to generate dem terrain directly to a single tile?

I haven't tried a 4m mesh just yet but I'm going to try that out next.

This program, without exaggeration, is a game changer. It makes things so much simpler, and I can't express my appreciation enough, I'm sure many will feel the same way. Making it work worldwide means that it is going to get a lot of use. I kept up with your posts while you were developing Lidex and I was wishing it would work for the rest of us too...

Regards

Tony
.

Hi Tony,

Thanks for the very kind words—much appreciated, sir.

One of my primary goals with LIDEX is to make route building far more approachable for first-time route builders while also eliminating hours of repetitive preparation for experienced builders.

Anyone who has fought DEMEX out-of-memory errors, searched for the correct GeoTIFFs, converted them into something DEMEX could use, or followed the rather convoluted process required to create Distant Mountains will appreciate the difference.

LIDEX turns that entire job into a largely point-and-click operation. Give it the route coverage—through an existing route, tile list, marker file, KML, or track database—and it can create an entire route’s terrain mesh and Distant Mountains in one controlled operation. It also handles terrain-source selection, downloading, projection, conversion, tile seams, and reuse of previously downloaded data.

--------------------------------------------------------------------------------

As for satellite or aerial imagery, I have not explored that area very deeply in either LIDEX or TSRE GenX yet.

That is separate from the OSM map system currently in LIDEX.

For anyone unfamiliar with it, OpenStreetMap data is not a collection of finished map images. It is a huge vector database containing nodes, lines, polygons, and relations. Those features carry tags describing what they represent—roads, railways, waterways, forests, farmland, buildings, and so forth.

LIDEX reads that raw data, projects it into the route’s coordinate system, classifies the features, assigns colors and drawing priorities, and then renders the reference map textures used by the route editor.

The LIDEX and TSRE maps should look quite similar. That is intentional, as I reproduced TSRE’s OSM drawing palette and general rendering recipe inside LIDEX. The important difference is that LIDEX can obtain and process the OSM data for the entire route in one pass rather than working one tile at a time.

LIDEX also retains the underlying vector information and creates the route-wide derivatives used by the new PolyVeg system in TSRE GenX.

--------------------------------------------------------------------------------

I would not recommend using the 4-meter mesh for a production route just yet.

The current public releases of TSRE and stable Open Rails do not fully support it. My development version of TSRE GenX is now operational with the 4-meter mesh—to the best of my knowledge—and I intend to include that support in the next release.

Open Rails Unstable also has initial support, thanks to Eric and James, but a few issues remain. In particular, some builds still calculate terrain-lighting normals using assumptions inherited from the standard 8-metre mesh. That can produce false shading even when the 4-meter terrain geometry itself is correct.

I have successfully loaded and driven a 4-meter route in Open Rails, and the overall performance appeared comparable to the traditional 8-meter mesh on my system. That is encouraging, but one successful route is not enough to declare the feature production-ready.

For now, please consider the 4-meter option experimental, use it only on a test route, and keep good backups.

--------------------------------------------------------------------------------

I am genuinely pleased that you find value in this work. Feedback like yours helps justify the countless hours that have gone into LIDEX and TSRE GenX—and reassures me that I am solving problems route builders actually want solved.

Thanks again.

Regards,
Scott

[Tonya37]

Hi Scott, your comments explain a lot! By the way, (and just for example) there are available hires DTMs available locally in places. I laid a route many years ago (using MSTS RE) based in the Italian region of Friuli Venezia Giulia using 1 second data publicly available from the regional government, with compatible 1:5000 mapping. I used Demex to load the terrain, and the maps were placed as transfers (the maps showed every track in detail). The result was stupendous, no terrascaping was required. I simply placed the tracks to follow the ground. But it took a huge amount of work even for that short route (about 20 km). Modelers in the US are fortunate to have hires DTM so readily available, thanks to LIDEX. I look forward to getting into Lidex and to future developments. I think that the developments made by you, Eric and Goku could well lead to a rejuvenation of the hobby.

Regards

Tony

[Tonya37]

Me again! I just wanted to add some comments about the accuracy of the 30 m Copernicus dem. I would have included a picture but still don't know how! I added an overlay (prewar British OS map) to a downloaded map tile, then used lidex to create the topography. I ended up with some very pleasing results:
1. The topography conformed extremely well with the map.
2. The map had a few benchmarks shown (from about 1945 - 1950, I think). I located 7 and compared them with the terrain elevations. Three of them agreed to within 0.1 m. Another three agreed to within 1 m, and a third to within 2m. Given the stated accuracy of the dem, this is excellent agreement.
3. The crests of some cuttings I made (using the 'F' key) agreed very well with those shown on the map.
4. Quite clearly the default offsets used by Lidex are spot on (unlike the dems I was using previously. Those were out by meters, but I had no way of obtaining accurate offsets nor of applying them,

Bottom line is, even using the Copernicus data, the result is still very accurate and is really satisfactory for most uses. Lidar is brilliant, but for most routes little is lost by having to use Copernicus. We just have to try a little harder when it comes to track laying

Regards

Tony

[scottb613]

Me again! I just wanted to add some comments about the accuracy of the 30 m Copernicus dem. I would have included a picture but still don't know how! I added an overlay (prewar British OS map) to a downloaded map tile, then used lidex to create the topography. I ended up with some very pleasing results:
1. The topography conformed extremely well with the map.
2. The map had a few benchmarks shown (from about 1945 - 1950, I think). I located 7 and compared them with the terrain elevations. Three of them agreed to within 0.1 m. Another three agreed to within 1 m, and a third to within 2m. Given the stated accuracy of the dem, this is excellent agreement.
3. The crests of some cuttings I made (using the 'F' key) agreed very well with those shown on the map.
4. Quite clearly the default offsets used by Lidex are spot on (unlike the dems I was using previously. Those were out by meters, but I had no way of obtaining accurate offsets nor of applying them,

Bottom line is, even using the Copernicus data, the result is still very accurate and is really satisfactory for most uses. Lidar is brilliant, but for most routes little is lost by having to use Copernicus. We just have to try a little harder when it comes to track laying

Regards

Tony

Hi Tony,

I am truly happy it seems to be working as advertised and I appreciate your enthusiasm - getting everything to line up took a bit of doing. My early versions of LIDEX were a bit off - and it required considerable effort to hunt down the root cause. My testing was limited to the Northeast US - as a single person I do not have the time nor the energy to comprehensively test globally. I did have one small route as a "global" test in the Austrian Alps - the Semmering Railway. After I applied the fix below - EVERYTHING seemed to line up perfectly. When you use 1m LIDAR on 4m HD terrain mesh in the US - every bridge underpass is very clearly defined which helps validate alignment. LOL - basically - it was a "Hail Mary" for the remainder of the globe to line up. Thanks for confirming it appears to be working well in other locations.

SCO LIDEX - Default Terrain Geometry Placement Correction
================================================== ========

Observed problem
----------------
Terrain generated on several default-projection routes consistently needed
an east/south correction to line up with the TSRE/OpenStreetMap route overlay.
The coordinate comparison resolves that correction to approximately 11 meters
east and exactly 16 meters south.

The 16-meter north/south error is exactly two normal ORTS terrain-post
intervals. The east/west correction is not an exact 8-meter post multiple and
is most accurately applied while sampling the source DEM.

What was checked
----------------
The normal terrain grid itself is positioned correctly:

* The raw grid contains 256 by 256 stored posts.
* Post spacing is 8 meters.
* Post (0,0) is at the northwest tile corner: X=-1024, Z=+1024.
* Stored posts cover 255 intervals, or 2040 meters.
* The east and south boundary posts are supplied by neighboring tiles.

Therefore, the 2040 meter stored span inside a nominal 2048 meter tile is
intentional and was not the source of the placement error.

Root cause
----------
The older Open Rails WorldLatLon converter and TSRE's map-coordinate
converter use slightly different Interrupted Goode Homolosine image origins.

Older ORTS constants used by SCO LIDEX:

Upper-left X = -20,013,965
Upper-left Y = 8,674,008

TSRE map converter constants:

Image left = -20,015,000
Image top = 8,673,000

After the different tile-index formulas are accounted for, the north/south
difference is exactly 16 meters. The calculated east/west difference is
approximately 11 meters. SCO LIDEX now keeps that calculated value instead of
rounding it to a complete 8-meter terrain-post interval.

Implemented correction
----------------------
For routes using the standard MSTS/Open Rails projection, SCO LIDEX now
applies a built-in terrain placement correction of:

11 meters east
16 meters south

The correction is applied at geographic sampling time. Every selected normal
terrain tile and Distant Mountain tile uses the same corrected geographic
frame, preserving seamless transitions.

Routes containing TsreGeoProjection are not given this standard-projection
correction. They continue to use their route-specific TSRE projection.

The Advanced Geo Bias controls remain available. Their values are additional
to the built-in standard correction and should normally remain at zero.

Source references
-----------------
Open Rails:
Source/Orts.Common/WorldLatLon.cs

TSRE5:
GeoCoordinates.h
GeoCoordinates.cpp

SCO LIDEX:
Program.cs
GeoTileMapper

Regards,
Scott

[Tonya37]

For the record the figures I recorded are for NW England. The plan positions of the benchmarks were accurate - I aligned the overlay 1945 OS map with the TSRE5 generated map tile. This does illustrate the usefulness of being able to apply overlays from printed maps.

Tony

[demando]

Not sure if it's a bug, but when I try to run 1401, it runs for a second then disappears. I haven't been able to run SCO LIDEX since 1.100.
Edit: It only works if it's sent to the desktop, which I would prefer free of clutter. Is there a way around this to allow the exe to run from the install folder, Scott?

[scottb613]

Not sure if it's a bug, but when I try to run 1401, it runs for a second then disappears. I haven't been able to run SCO LIDEX since 1.100.
Edit: It only works if it's sent to the desktop, which I would prefer free of clutter. Is there a way around this to allow the exe to run from the install folder, Scott?

Hi S. J.,

This is a new one - thanks for reporting it.

Since LIDEX closes before actually starting, it never gets far enough to create a log. Windows 10 or newer is the supported OS.

Please delete the entire old SCO LIDEX program folder—do not extract the new release over the existing files. Download v1.401 again and extract the complete folder to a normal local location. Do not move or copy the EXE by itself, and do not run it from inside the ZIP.

To create the shortcut, the preferred method is to run the shortcut-creation AddShortcutDesktop.cmd file included with LIDEX in the root directory. This keeps you out of the program files and creates a shortcut pointing to the correct location.

Alternatively, right-click SCOLIDEX.exe and select Send to → Desktop (create shortcut).

In either case, once the shortcut has been created, you can move it anywhere you want—it is not locked to the Desktop. You can also pin it to Start or the taskbar.

If LIDEX still disappears, please tell me the exact folder path from which you are running it and whether Windows Security shows anything under Protection History.

Regards,
Scott

[scottb613]

Not sure if it's a bug, but when I try to run 1401, it runs for a second then disappears. I haven't been able to run SCO LIDEX since 1.100.
Edit: It only works if it's sent to the desktop, which I would prefer free of clutter. Is there a way around this to allow the exe to run from the install folder, Scott?

Hi S. J.,

Oh - here's a thought - my Malware Bytes had a false positive that it was Ransomware and quarantined my executable while running it - making the program disappear mid run. I had to make an exclusion for the application directory in MB. I think it was because I am an unknown developer with a program that does large file manipulations - which triggered it. There is some process to register applications MS offers to certify programs - this being a hobby and free for a very small community - I don't think I am going down that route. While I do make every attempt to make sure my programs are safe and easy to use - it is definitely "use at your own risk" - and my source code is published for public record.

Regards,
Scott

[demando]

Thanks Scott. I'll have to check my randsomware software to figure out if that is happening, maybe make an exclusion for it. I've got to wait a while, since SCOLIDEX is running right now through the desktop shortcut - going forward, this makes sense.