SRTM data conversion to Falcon 4 Terrain format

[Sherlock]

As most of us already know the SRTM data is available from the shuttle missions. As I understand it, this is 90 meter resolution elevation data as opposed to the 250 meter DCW elevation resolution data. We know the SRTM data can be converted for use in Falcon based on Blueprints and Zaggy's remake of the Korean theater (BAZT SRTM Korea 1.5 installer).

I would like to learn more about how to do this. What is the process? Where do I start? Is this feasible for remaking some of the theaters on PMCs plate now like ODS, Europe, etc?

Would Zaggy and Blueprint be willing to share their work and knowledge?

These are the types of questions that I am thinking about....

Any help would be appreciated...

[Snake Man]

SRTM data is available from the shuttle missions.

Yeah, but where, do you have a link?

this is 90 meter resolution elevation data as opposed to the 250 meter DCW elevation resolution data.

Umm really? I thought that its 250m as the NASA DEM we use is 1000m.

Well I know the dem is 1km, dunno about the SRTM.

Is this feasible for remaking some of the theaters on PMCs plate now like ODS, Europe, etc?

I would be very interested about doing this detailed terrain data if possible. If I'm correct, all it needs is more accurate data which is then put into the L2. No tiling or otherwise need to be fixed, except maybe on special cases on coastlines or so. But this is... just guessing

[marvin157th]

SM,

That would be a great idea for the Afghanistan theater as it is so mountainous. Plus the tiling is early beta stages, so it would not be anymore work really.

Just a thought

Jody

[T_Rex]

MadDog has outlined potential ways to use GoogleEarth elevation data in F4, but I'm not sure if he's worked up a real workflow for it.

Oh and Marvin - nice to see you around again.

[Sherlock]

SRTM data is available from the shuttle missions.

Yeah, but where, do you have a link?

this is 90 meter resolution elevation data as opposed to the 250 meter DCW elevation resolution data.

Umm really? I thought that its 250m as the NASA DEM we use is 1000m.

Well I know the dem is 1km, dunno about the SRTM.

Is this feasible for remaking some of the theaters on PMCs plate now like ODS, Europe, etc?

I would be very interested about doing this detailed terrain data if possible. If I'm correct, all it needs is more accurate data which is then put into the L2. No tiling or otherwise need to be fixed, except maybe on special cases on coastlines or so. But this is... just guessing

This site looks very promising... http://srtm.csi.cgiar.org/

I haven't looked at the data yet though.

[lamurt]

i've been searching for the SRTM data since i posted in the theather tools section and believe me there are tons of info on the subject,, however finding the useful ones seems to be a work in itself, anyway here's a link worth checking for:

http://edc.usgs.gov/products/elevation.html

Seamless Data Distribution System is the interesting one here, there is also a GTOPO30 link down there but i believe that is what we are already using, which is the 1km dem data.

[lamurt]

disregard the gtopo30 (getting older... ), i meant this:

http://glcfapp.umiacs.umd.edu:8080/esdi/index.jsp

[Closter]

Hi all!

This is one of my subjects of interest, too. Some wrong assumptions here, as I see it.

-The data in L2 is from the 30 arcsec type, so along the N-S direction is about 900 meter each vertex. Along the E-W, it depends on the latitude, but always is less or equal.

-The data in L1 and L0 are derived from the L2 through SPTInstall, in a bilinear interpolation way. Kind of overkill, in my view. We don't need data that we can't see. But luckily, we can alter these two grids to improve the detail. That`s what BaZ did. So they are a 15 and 7.5 arcsec grid.

-The data from SRTM is 1 arcsec in some parts of the world (US mainly) and 3 arcsec in the rest. So 30m and 90m along the N-S. Pretty much for us (not bad at all).

-This data is available in 1 degree x 1 degree chunks from a FTP server, in a simple binary format.

-The only problem is to get this data into the Lx, Ox files.

-I would love to know these formats, Lx Ox. Please, some info here.

[Closter]

Here it is some examples (Theoretical ) of the gains that we can obtain from the srtm.

This is the level of detail that comes out of L2:





This is the detail (bilinear interpolated) that is made in L0 from the L2 through SPTInstall.




And this is what we could obtain if we use the srtm 90m (3 arcsec) data and convert them to 225m (7.5 arcsec) and then insert them in L0.





It's pretty cute, ¿isn't it?


Well, this is a simulation over photoshop, but the data I've used are real data from our theater.

Another issue is that to view this details, it is needed that the LOD, "switches on". So we must fly under some altitude, or we must bump the graphics slider in the setup dialog.

[Sherlock]

Hi all!

This is one of my subjects of interest, too. Some wrong assumptions here, as I see it.

-The data in L2 is from the 30 arcsec type, so along the N-S direction is about 900 meter each vertex. Along the E-W, it depends on the latitude, but always is less or equal.

-The data in L1 and L0 are derived from the L2 through SPTInstall, in a bilinear interpolation way. Kind of overkill, in my view. We don't need data that we can't see. But luckily, we can alter these two grids to improve the detail. That`s what BaZ did. So they are a 15 and 7.5 arcsec grid.

-The data from SRTM is 1 arcsec in some parts of the world (US mainly) and 3 arcsec in the rest. So 30m and 90m along the N-S. Pretty much for us (not bad at all).

-This data is available in 1 degree x 1 degree chunks from a FTP server, in a simple binary format.

-The only problem is to get this data into the Lx, Ox files.

-I would love to know these formats, Lx Ox. Please, some info here.

Are you a programmer? Can you create a utility to convert the data? I have the Lx/Ox formats somewhere...I just have to find them.... Will edit this post when I do.

------------------------------EDIT------------------------------------------------
Here are the notes I have accumulated on this subject along with the links I picked them up at. Not sure if the FF link will work now that they have a new forum:

http://www.freefalcon.com/forum/viewtop ... aggy#73082

File formats are as follows:

O2 file: 4Bytes per Record
DWORD(LONG) - L2 byte Location

(This file essentially, represents the SEGEMENTS of the theater, starting a 0,0 in the LOWER LEFT corner, to 63,63 in the UPPER RIGHT. Each of the DWORD(LONG) Records points to the location in the L2 file (in BYTES), that the 16x16 Tiles that make up each segment, begins. Like the O2 files, the order of the 16x16 segement records in the L2 is, starting at Lower Left, going to Upper Right)

L2 File: 9Bytes per Record
DWORD(LONG) - Texture Offset Number
WORD(INTEGER) - Elevation
BYTE - Palette
BYTE - X1
BYTE - X2

(Bytes 1-4 hold the Texture info, this what you want! 5-6 hold the elevation (in ft), 7, palette, 8 and 9 have something to do with the Normals, and you dont need to worry about them either!)

-So now you have an array of offset numbers (representing tiles)
-now you need to use the Texture.bin to figure out the Offset Numbers of the tiles and related them to the Filenames (im not TOO familiar with this format, but basically, the tiles are arranged into tilesets, with are 16 tiles long. So if there are 16 tiles in that tileset, you number them 1-16... If there less than 16, you still assign 16 numbers to that tileset... For example, if you have a tileset that only contains 8 tiles then a tileset of 16 tiles, the numbers would look like; 1-8 (for the first tileset) then 17-32 for the second tileset... I hope that helps)

-----------------------------

http://www.freefalcon.com/forum/viewtop ... rmat+zaggy

L2 Record
2-byte
===============================
WORD Texture;
WORD Elevation;
BYTE Palette;
BYTE X1;
BYTE X2;

4-byte
===============================
DWORD Offset;
WORD Elevation;
BYTE Palette;
BYTE X1;
BYTE X2;

WORD = 2 bytes
DWORD = 4 bytes
BYTE = 1 byte

O2 Record
2-byte & 4-byte
===============================
long (4 bytes) Pointer to each Section in L2.

Section = 16 x 16 x size of L2 record

And looking at that structure then, the only thing a 4byte structure does, is allow a larger number of textures to be used? Not that I want to mess with that area of it... Just the elevation stuff... And that shouldnt require messing with the O2, should it?


The L2 Offset points into fartiles.raw. With the more diversified tiled theaters, like the Balkans, the 16-bit offset was limited to 64,767 fartiles (Blkans has ~102,000 fartiles). Making the Offset a 32-bit integer extended the number of fartiles to 4+ gig tiles.

The O2 offsets (pointers) change due to the size (bytes) of a L2 section.

[Sherlock]

MadDog has outlined potential ways to use GoogleEarth elevation data in F4, but I'm not sure if he's worked up a real workflow for it.

Oh and Marvin - nice to see you around again.

Thx T_Rex...yeh, he's moved all his nice pics so now to get the full impact of his thread you have to go to their website at combat360 here:

http://www.combat360.org/forum/topic.as ... hichpage=2

You have to be registered with the site to see the forum thread.

[marvin157th]

Oh and Marvin - nice to see you around again.

TRex,

Thanks dude! It would appear as though many names are back! Great things are yet to come!

Jody

[Sherlock]

@ Closter,
I updated my post above with all the Lx/Ox format information I have in my notes. Can you use that ?

[kinggeorge]

New elevation data could be the same milestone as high res textures, i fly srtm terrain in the university research sim from time to time and its amazing even if the textures are boring. Sense of speed is much better!
Problem is the terrain textures won't match - they are 1000x1000m - point. Most srtm data should be available on the net somewhere i'd think - but not everything is in 90x90, 250m would also be a good improvement.
At least it is absolutly worth checking out the options. I remember someone of the community messing with matlab to get the data straight?

[Closter]

I updated my post above with all the Lx/Ox format information I have in my notes.

Wow, that was... impressive.

Are you a programmer?

No, sir. I am engineer and have some programming background, but I prefer to use the tools available if we could avoid having to mess with that.

I am thinking of using BaldEagle's last LxNormalFix, if we could translate the files to the format that it accepts. Then we'd have got heights and normals at once (with the annoying fog issue...). Better not working twice!.

So I am trying to find some script to translate our .hgt files into .ded that LxNormalFix accepts. I suppose that program needs a .ded from the whole theater, so we will have to join all the data into one big .ded and then use it with LxNormalFix...

Another way could be joining the data in raw like an image and export it as greyscale bmp. But the programs I use cannot handle the 16bit greyscale properly, and I am not sure if the LxNormal Fixer accepts this kind of bmp, or simpler 8bit deep.

Or perhaps there is a parameter to make LxNormalFix to accept partial elevation files, that we don't know for the moment.

Perhaps BaldEagle would be kind enough to make this available... plzzzz!

But if necessary, we could use some code as base, like dem2lf.pl or something like that.

Food for thought.

[Closter]

Hi! Just a quick update about my findings on this issue...

This is a relief map with the data above: it resembles better to the human eye the shape of the terrain, just simulating the shading due to the sunlight.



Do you like it?

[Sherlock]

Hi! Just a quick update about my findings on this issue...

***SNIP***

Do you like it?

Me LIKE IT! Got more info to share?

[marvin157th]

Want to make sure I understand this.. If you can some how make the terrain look like that with shading, that will be so cool!!!!

Specially for desert mountaneous theaters such as Afghanistan. I flew FSX a few months ago, and I noticed that is almost exactly how they made their desert terrain look so good. It was just one colour with tons of shading like that, and it looked very real.

Hope we can get that working!

Jody

[Closter]

The square terrain that you see is actually a rectangle 88 km wide and 111 km high. So I think is too big for what you can appreciate easily in Falcon. The geometric detail in Falcon, in the usual terrains, is limited to 250 meter. For the effect to be as good as in FSX we'd need to use the whole SRTM resolution about 90 m., and the shading algorithm that FSX uses.

The textures are employed not only to give colour but to fake the eye about the geometric detail we have available. So we must use the best combination.

Given the somehow fractal look of the earth surface, we could use this approach to make beautiful tiles, changing the scale. Perhaps this will achieve the effect you are looking for.

[Zaggy]

You guys should try looking up Blueprint, on FW...

He and I did a LOT of work with SRTM for BaZT stuff...

[Snake Man]

I think I'll ask him, however you're here and now... how about telling us what you can?

[Zaggy]

Well, what is there to say? Bluey, last i was atlking to him, was close to getting a full SRTM theater creator app made...

Me on the other had, I was the one coding interpolation algorithms, so we can intelligently add data to existing theaters, from the base L2's...

Bluey's method is BETTER, however required a lot of work... Im not sure if he has worked out how to exactly synchronise e00's with the DEM, but as i said, its been ages since we talked, me getting all smashed up and such. His method is great for theaters going from scratch...

My method was an attempt to enhance existing theaters built from the 30ArcSec Data, buy using some algorithms i wrote that analysis the shape of the terrain, and introduce new mid-way points based upon the shapes round them (ie, it takes two steps... you load your L2, then it is interpolated upto 500m data for the L1, then again to 250m for the L0)... My hassle was repacking the L1 and L0 with corrected normals...

It was pretty nasty HACK code too... I must recode the algorithms one day tho!

[Couby]

Well, what is there to say? Bluey, last i was atlking to
My method was an attempt to enhance existing theaters built from the 30ArcSec Data, buy using some algorithms i wrote that analysis the shape of the terrain, and introduce new mid-way points based upon the shapes round them (ie, it takes two steps... you load your L2, then it is interpolated upto 500m data for the L1, then again to 250m for the L0)... My hassle was repacking the L1 and L0 with corrected normals...

So could it be applied to existing terrains we have ? Like ODS, Iran, Taiwan, etc...

[Closter]

Hi, Zaggy, glad to see you overhere! You and Blueprint did a great work with your BaZT saga.
As you can see, I am interested in this issue, and I have made my own research. I will be happy to help you (Blueprint and you) on that super-theater-app. No need for several developments if we can make one-the best!

I will send you my e-mail address via private mail.

[Zaggy]

Couby: If i find time with school, it could be written to work with older theaters, yes... Im doing my Major Project this semester, so time will be limited until afterwards... But yeah, i was designing (and coding) the original app to be able to work with legacy theaters, and not JUST new build..

The original BaZT releases used this intelligent interpolation technique...

[Zaggy]

Actually, If i find time, i will find the early (simpler) versions of the algorithm, type it up and post it... Someone else maybe able to turn it back into code faster than i, at the moment...

[Couby]

Great

[87th_striker]

Good to see you back, Zaggy !!!

[Sherlock]

Good to see you back, Zaggy !!!

X2...get well soon m8!

[Blueprint]

What can be done for existing theaters?
If they have been created with the dem2terrain tool, it's not possible to update it with SRTM data because there is no way for me to recreate the projection dem2terrain performs (I spent several weeks trying). The only option I see is to use Zaggy's interpolation approach to create more rough terrain on the existing data. The alternative would be to create the theater terrain and objective data from scratch based on SRTM.

What can be done for new theaters?
The theater creation tool I was working on for Europe never really got finished but I think most of the functionality I wanted made it in. The tool takes SRTM-3 data and creates L2/O2 files for it. Then it uses a global feature dataset to create tdf files for use with cate and an objectives.csv for tacedit. At this point all the other theater creation tasks can take place eg. tiling, campaign creation. When the theater is finished, an installer needs to be built to properly write the L1 and L0 elevations into the theater files (just like the SRTM Korea Installer).

If there is a strong desire, I might go about creating the files for a new theater based on my personal (fixed) SRTM-3 dataset.

[ranger822]

Hum . . . there is strong interest (at least on my part) to rebuild Israeli Theater with SRTM. I can't do the whole thing myself. But I would be glad to be part of a working team.

IMHO any SRTM is probably like socks on a rooster if you don't significantly improve the terrain tiles to go with the new terrain. Moreover, to really justify the work, which I am guessing would be significant, the end prodcut ought to represent a significant visual improvement over what exists.

I would seriously consider undertaking the project if a few things were in place -

1. Someone to help with tiling/CATE,
2. Someone to help with making new tile sets to really give Israeli theater a much better look, perhaps photorealistic look in selected areas.
3. A demonstrated ability to build new working airbases per another PMC thread.

Anyone else out there seriously interested?

[Snake Man]

Thanks for the information Blueprint, appreciated you came here to explain it. Altough I'm sad to hear Dem2Terrain spherical method is not usable.

I don't know if I'm too old schoold but if we have to scrap dem2terrain (method), then I'm afraid I'm not that interested anymore. We worked hard to get the spherical thing going on and I'm not willing to let it go

Not to mention that many (all) theaters would have to be remade, thats just simply something I cant do right now. Even Iran which just got started, with the linking is already so far that it would be more trouble than worth to scrap and start from scratch.

[T_Rex]

You guys might think I'm crazy, but I think there's value in having both methods available.

There are plenty of other theaters to be made, that haven't been started yet, that could possibly benefit from the SRTM. And, IMHO, options are good.

[ranger822]

I am in agreement with Tom on having the options. If it isn't a major chore, I would like to have the tool available. If someone is willing to bust their gut making it then there is just as likely someone that will use it. There is always a lone wolf type like Twister and Dudi that can do it all. They can do it and probably will if the tools are there. Not having the tool just closes the door. We don't know what kind of other things will be developed which might make current obstacles a one-click process.

So much effort has been put into several theaters now - it seems a shame to write them off for SRTM just because of an itsy bitsy thing like having to remake the terrain and link it all back up

[ccc]

ya agree with T.

for existing theaters we need Zaggy's tool.

for brand new theaters we can use NEW SRTM tool.

[Snake Man]

Yes of course by all means do the thing/tool etc.

All I'm saying is that the PMC theaters most certainly wont be remade. I already did this once with ODS and Vietnam, that is enough for me. ODS for example now has had countless man hours poured into it and I wont let them go, sorry but thats how things are.

But please do the SRTM work and I'll support it on tutorials and host the tools etc whats required 100%, no problem.

[Malc]

Does an SRTM projection give the same projection as DEM2L2? If it does, perhaps we can look at a small tool to read the tile offset info from existing theaters and transfer it on top of the new SRTM data? New terrain altitude data with old tiling work! Perfect!

Hope that makes sense!? 8)

[T_Rex]

That makes ALOT of sense, actually.

[Closter]

t's not possible to update it with SRTM data because there is no way for me to recreate the projection dem2terrain performs (I spent several weeks trying)

I think I have read some place, that the projection in Falcon is not a standard one. But from the use of the tools we can see that the latitude is not changed. All vertexes in a horizontal line have the same latitude. So they are located in a paralel. So the point to me is the correction of the differences in longitude.

I think I have some ideas... PM incoming!

[Closter]

Pay attention to this:


And this(they are not the same theater, it doesn't matters):

Theater name: alaska3
Theater size: Regular
Size (L2 segments): 64 x 64
Size (L2 tiles): 1024 x 1024
Size (Nautical Miles): 552.377 x 552.377
Resolution: 0.134858 Nautical Miles
Centered on: Lat 71 Long -159
NW: Lat 75.6031 Long -177.514
NE: Lat 75.6031 Long -140.486
SW: Lat 66.3969 Long -170.496
SE: Lat 66.3969 Long -147.504
Highest point: 7639ft
Lowest point: 0ft